EVE Search - Proposed Changes to the Static Data Export (SDE)

All Channels

EVE Technology Lab

Proposed Changes to the Static Data Export (SDE)

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 3 4 :: [one page]
Author	Thread Statistics \| Show CCP posts - 17 post(s)

CCP Tellus C C P C C P Alliance 36	Posted - 2016.02.10 18:24:23 - [1] - Quote Hello everyone, We know that some of you have been frustrated with late SDE releases and last-minute-changes in the past. Building and deploying the SDE has always been a manual process that had a tendency to slip through the cracks. We want to build and ship the SDE to you all sooner, and more often alongside both Tranquility and Singularity releases. This would necessitate some changes to the SDE and its build process in order to achieve continuous delivery. In the process of migrating static data from our legacy BSD content authoring tool to our new FSD authoring tool; we've been moving away from editing tables in a centralized SQL Server database, to YAML files tracked in revision control. This process is reflected by the SDE, having shipped both a SQL Server backup file and YAML files for some time now. We don't anticipate this migration to finish anytime soon. One way of simplifying the build process can be achieved by naïvely converting the SQL tables to YAML files. As an example, the last two entries of the eveUnits.yaml file would look like: - unitID: 141 unitName: "Hardpoints" displayName: "hardpoints" description: "For various counts to do with turret, launcher and rig hardpoints" - unitID: 142 unitName: Sex displayName: "1=Male 2=Unisex 3=Female" description: null The first entry of staStations.yaml would look like: - stationID: 60000004 security: 0 dockingCostPerVolume: 0.0 maxShipVolumeDockable: 50000000.0 officeRentalCost: 10000 operationID: 26 stationTypeID: 1531 corporationID: 1000002 solarSystemID: 30002780 constellationID: 20000407 regionID: 10000033 stationName: "Muvolailen X - Moon 3 - CBD Corporation Storage" x: 1723680890880.0 y: 256414064640.0 z: -60755435520.0 reprocessingEfficiency: 0.5 reprocessingStationsTake: 0.05 reprocessingHangarFlag: 4 In order to smooth out such a transition, we would of course ship the good o' SQL Server backup file and the SQLite universe data alongside these new YAML files for the next one or two SDE releases for Tranquility. That way you'd have time to update any tools you may be running on top of the Static Data Export. What are your thoughts and opinions on this proposal? Thanks!

Desmont McCallock 598	Posted - 2016.02.10 19:48:55 - [2] - Quote tl;dr Get done with the transition and give as one format and make it an MySQL dump one. Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats. I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE. And most like this format would be an MySQL dump. From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them.
Florian Lousberg Furtherance. 2	Posted - 2016.02.10 20:22:21 - [3] - Quote I personally would welcome getting all the SDE tables in plain YAML on a more regular basis from an official source. For me it would be much easier to handle than waiting for FuzzySteves MySQL conversion (thanks for that, Steve! You got my vote!) and then converting that again into the custom Django database schemes I use in my projects. Besides, the ability to just run a plain text diff against the previous version would be really nice, too.
Zifrian Licentia Ex Vereor Phoebe Freeport Republic 1707	Posted - 2016.02.10 21:01:54 - [4] - Quote As much as I dislike yaml, I'd rather you just publish the entire SDE in yaml for a few cycles along with the SQL server file and then phase the SQL Server file out for good. Because what I'm going to do is convert the data into what I want to use and then take only the data I need for my app. It would be great if you provided a single DB format (SQLite like the universe data?) that we could use but I don't care as long it's all the same. If yaml works for you best, then do it in that. If I read your post correctly, it sounds like you want to add just two more files to yaml? Many of us have been dealing with this hodgepodge system for 4 years now and I'd just like you to get on with it and do it instead of this piecemeal approach. If you are going to release a few files here and there, then that is 100xs more frustrating to me than the current system and I'd rather you do nothing instead. GÇ£Any fool can criticize, condemn, and complain - and most fools do. GÇ¥ - Dale Carnegie Maximze your Industry Potential! - Download EVE Isk per Hour!

CCP Tellus C C P C C P Alliance 37	Posted - 2016.02.10 21:10:02 - [5] - Quote Zifrian wrote: If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files. One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour. There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes.

Zifrian Licentia Ex Vereor Phoebe Freeport Republic 1707	Posted - 2016.02.10 22:05:28 - [6] - Quote CCP Tellus wrote: Zifrian wrote: If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files. One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour. There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals. OK that's great then. I'm glad it's going to be all at once. Many of us have wanted this for years and totally understand your reasons for doing it. Will the universe data also be migrated from SQLite? Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live? Thanks for the update, looking forward to this. GÇ£Any fool can criticize, condemn, and complain - and most fools do. GÇ¥ - Dale Carnegie Maximze your Industry Potential! - Download EVE Isk per Hour!

CCP Tellus C C P C C P Alliance 38	Posted - 2016.02.10 22:22:30 - [7] - Quote Zifrian wrote: Will the universe data also be migrated from SQLite? The universe data is authored as YAML files internally. These are then post-processed into an SQLite database. Why? I am not sure exactly. I'm contemplating just removing that post-processing step. Zifrian wrote: Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live? That's the idea! Automatically building SDE exports for Singularity as soon as changes are deployed there shouldn't be much trouble either with these changes.

Max Kolonko WATAHA. Fidelas Constans 588	Posted - 2016.02.10 22:51:17 - [8] - Quote As long as some wizard like steve will provide his service of converting them to something old dog like me can understand I'm fine :) Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Dragonaire Here there be Dragons 78	Posted - 2016.02.11 00:20:55 - [9] - Quote Not that I deal with the SDE directly to much but having everything in a single format should make dealing with it easier IMHO. Yaml is a good format for doing other conversions, from what I've seen and depending on the app it might be usable directly too. I would think Steve's conversion tool would also be easier to maintain and automate with just the one format. Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5875	Posted - 2016.02.11 00:57:13 - [10] - Quote Go for it. My conversion will be a lot simpler. And easily transferable to other people. (i.e., if you have a fairly basic python install, you'll be able to import it. Into any database that sqlalchemy supports.) Granted, it's going to be one big import file (probably) and it'll take a while to import, but it's a one off job. And I can do it directly on the server. Woo! CSM X! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Masao Kurata Perkone Caldari State 350	Posted - 2016.02.11 01:03:45 - [11] - Quote YAML as the one true format sounds great, doing conversions yourself is too painful otherwise.
Pete Butcher KarmaFleet Goonswarm Federation 310	Posted - 2016.02.11 05:28:20 - [12] - Quote Do you plan to convert mapDenormalize too? That would be one big ass YAML http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Mr Mac Dark Goliath 124	Posted - 2016.02.11 06:46:33 - [13] - Quote I like less YAML but having entire SDE in single format is fine.
Desmont McCallock 598	Posted - 2016.02.11 08:22:19 - [14] - Quote I agree with Steve. Our tools would have more meaning (to the linux savvies out there) if we could get rid of the SQL Server. In that case I could even make my tool target .NET Core and have it running on every OS. On another note, as Pete already mentioned, what about the mapDenormalize table? This will produce the largest YAML file. Do we have a size number so we know approximately how big it's going to be?
Evanova Android Traquenard Labs 25	Posted - 2016.02.11 13:13:05 - [15] - Quote I can only speak for myself but so far, the YAML files have been an inconvenience more than anything; a pre-built SQL database is much easier for me to manipulate and use. I would guess that many applications will want to build a database out of the YAML files anyway, just to be able to create complex views out of many files/tables. Also, having to deal with it on a daily basis, I also dislike YAML very much and I think it is The Wrong Format for pretty much everything, but that's a personal and not very popular opinion. Now, in all practical sense, if exporting YAML files makes the data set more consistent, easier to update and simple to use and build things out of for CCP and datameister Fuzzy, that cannot be a bad change. Evanova - The Android App for Eve Online
BigSako Burning Napalm Northern Coalition. 120	Posted - 2016.02.11 13:17:19 - [16] - Quote Why do we need yet another format? The SDE should be in the same format as CREST, and that is JSON. JSON is easy to parse and you can directly use it in MANY applications, making (manual) converting unnecessary! edit: I would also like to see an index of the SDE, showing two things per "table" (file): Last Update + Update History / Diff
Pete Butcher KarmaFleet Goonswarm Federation 311	Posted - 2016.02.11 14:13:35 - [17] - Quote While we're at it, it might be a good idea to finally fix integrity problems within the SDE itself. There are relations to non-existing entities. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Shala Raan Trillionaire.pro 28	Posted - 2016.02.11 14:41:38 - [18] - Quote This looks nice. I use Steve's converions so if it helps him I'm all in (püñGÿ»ßù£Gÿ»)püñ TRILLIONAIRE.PRO
Erick1111 Quarantine Corporation 0	Posted - 2016.02.11 18:43:22 - [19] - Quote Having a one and only data format would certainly be a blessing. Then if we have conversions to make, we can make them internally.
Matt Faithbringer Ravens Of Faith and Light 29	Posted - 2016.02.16 16:18:05 - [20] - Quote Desmont McCallock wrote: tl;dr Get done with the transition and give as one format and make it an MySQL dump one. Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats. I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE. And most like this format would be an MySQL dump. From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them. I don't agree here. Not everyone uses MySQL. Some people use sqlite, some people use postgres. IMO it's bad idea to go with one format. Just migrate the whole thing to YAML, everyone can read that.
Matt Faithbringer Kill'em all. Let Bob sort'em out. 29	Posted - 2016.02.16 16:19:17 - [21] - Quote CCP Tellus wrote: Zifrian wrote: If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files. One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour. There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals. I'm not big fan of YAML, but having everything in one format would be awesome, even when that format is YAML.
Golden Gnu EVE University Ivy League 179	Posted - 2016.02.22 15:46:15 - [22] - Quote For me it doesn't matter directly. I will probably always use the conversion and/or tools by Desmont McCallock or Steve Ronuken. However, if you make their job easier, you're doing a good job. tl:dr I support whatever Desmont McCallock and Steve Ronuken says Creator of jEveAssets - the asset manager "Download is the meaning of life, upload is the meaning of intelligent life"
Mr Mac Dark Goliath 137	Posted - 2016.04.25 15:34:44 - [23] - Quote Patch notes 2016-04-25 Quote: Static Data Export (SDE): The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files.
Desmont McCallock 604	Posted - 2016.04.25 15:39:33 - [24] - Quote Mr Mac wrote: Patch notes 2016-04-25 Quote: Static Data Export (SDE): The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files. This is HUGE news. We need a pre-release asap.
Pete Butcher KarmaFleet Goonswarm Federation 324	Posted - 2016.04.25 18:20:51 - [25] - Quote mapDenormalize too? Holy crap, this must be a 1GB of yaml. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5978	Posted - 2016.04.25 19:19:41 - [26] - Quote https://cdn1.eveonline.com/data/sde/singularity/sde-20160416-2016-DANIELLE.zip is the prerelease version which has been floating round Slack. (be aware there's a couple of dodgy numbers in the blueprint data. it's not an export problem, just a problem with the source, which should now be fixed) I've got everything but the universe data loaded easily. Universe data shouldn't be too hard, but it'll need a chunk of processing, to make it look like the old data. (Fun story: mapDenormalize isn't how they store it at CCP. This is a cut from their versioning system.) Biggest differences so far: All the tables which were in the sql server export are now in single yaml files. loading them is just a case of iterating through the array and loading them back into a table. some tables are a touch troublesome due to the number of entries (invNames, for example), but most load just fine. ( https://github.com/fuzzysteve/yamlloader ) I'm running again for CSM 11, and I'd appreciate your vote. Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Desmont McCallock 604	Posted - 2016.04.25 21:36:22 - [27] - Quote Judging from Steve's linked SDE, universe data is split down to each individual solarsystem, which makes mapDernomalize table creation a process 3rd party tools have to undertake. BTW, thank you Steve for the link.

CCP FoxFour C C P C C P Alliance 4291	Posted - 2016.04.25 21:48:44 - [28] - Quote Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit. @CCP_FoxFour // Technical Designer // Team Tech Co Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Desmont McCallock 604	Posted - 2016.04.25 22:11:43 - [29] - Quote CCP FoxFour wrote: Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit. Feeewww. You got me worried there for some time.

CCP Tellus C C P C C P Alliance 57	Posted - 2016.04.26 13:07:47 - [30] - Quote There's a new SDE release available on our developes' site: https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY-legacy.zip

Desmont McCallock 604	Posted - 2016.04.26 14:23:10 - [31] - Quote @CCP Tellus Questions: 1. Will this be the naming format for the SDE filename from now on? 2. Where do we find the 'translationTables' table? Do we have to create it on our own?

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.26 14:45:14 - [32] - Quote Desmont McCallock wrote: 1. Will this be the naming format for the SDE filename from now on? Is there any particular reason you're relying on the filename format remaining consistent? It may or may not change in the future. Desmont McCallock wrote: 2. Where do we find the 'translationTables' table? Do we have to create it on our own? The 'translationTables' table was an internal look-up table used to construct the 'trnTranslations' and 'trnTranslationColumns' tables. It had not gotten deleted after the SDE export script ran. Do you know of a use-case for that table? Desmont McCallock wrote: 3. Would you mind fixing the duplicate entries in blueprints.yaml file? (Can provide exact info if needed) I can take a look at blueprints.yaml. Can you send me the details?

Desmont McCallock 604	Posted - 2016.04.26 14:53:18 - [33] - Quote CCP Tellus wrote: Is there any particular reason you're relying on the filename format remaining consistent? It may or may not change in the future. In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one) CCP Tellus wrote: The 'translationTables' table was an internal look-up table used to construct the 'trnTranslations' and 'trnTranslationColumns' tables. It had not gotten deleted after the SDE export script ran. Do you know of a use-case for that table? Mainly for backwards compatibility. I haven't ran into a situation nor seen anyone talking about that table, but you know how things goes. Remove something and someone will complain about it. CCP Tellus wrote: I can take a look at blueprints.yaml. Can you send me the details? Details already provided via editing my OP.
Salvoxia Evoke. Ev0ke 8	Posted - 2016.04.26 14:55:12 - [34] - Quote Hi, previous SDE releases always contained some kind of version or build number (for the last one it was 117575 for example). I always used it to reference the exact SDE release. Will there be something similar for the new release format or will you just use the release date? Best Regards, Salvoxia

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.26 15:55:08 - [35] - Quote Desmont McCallock wrote: In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one) . You can use the following regular expression: sde-(.+).zip We will probably not change the file naming scheme, but we may change where we host static data exports in the future. Salvoxia wrote: previous SDE releases always contained some kind of version or build number (for the last one it was 117575 for example). I always used it to reference the exact SDE release. Will there be something similar for the new release format or will you just use the release date? Use the date and the branch name.

Hel O'Ween Men On A Mission 159	Posted - 2016.04.26 16:05:28 - [36] - Quote CCP Tellus wrote: There's a new SDE release available on our developes' site: https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY-legacy.zip Is this upposed to be the official SDE release for the Citadel expansion. If so, ramTypeRequirements seems to have some PK issues: Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'. A duplicate key can't be inserted into the dbo.ramTypeRequirements object. The duplicate key value is (41590, 1, 38). EVEWalletAware - an offline wallet manager.

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.26 16:11:45 - [37] - Quote Hel O'Ween wrote: Is this upposed to be the official SDE release for the Citadel expansion. If so, ramTypeRequirements seems to have some PK issues: Am I missing something? I am not able to find a table called ramTypeRequirements.

Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5978	Posted - 2016.04.26 16:17:39 - [38] - Quote ramTypeRequirements is an old table. And it's probable that the 'bug' there is due to the conversion from yaml to the sqlserver db, with Desmont's code? I'm running again for CSM 11, and I'd appreciate your vote. Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Desmont McCallock 604	Posted - 2016.04.26 16:37:47 - [39] - Quote Yeah guys, it has to do with the duplicate entry in blueprints.yaml file I have already reported in this thread.
Desmont McCallock 604	Posted - 2016.04.26 16:43:39 - [40] - Quote CCP Tellus wrote: Desmont McCallock wrote: In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one) . You can use the following regular expression: sde-(.+).zip We will probably not change the file naming scheme, but we may change where we host static data exports in the future. I would rather avoid adding another regex search but you're the Boss.
Cwittofur Cesaille Quantum Distributions Crit My Hole 4	Posted - 2016.04.26 17:29:30 - [41] - Quote Desmont McCallock wrote: CCP Tellus wrote: Desmont McCallock wrote: In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one) . You can use the following regular expression: sde-(.+).zip We will probably not change the file naming scheme, but we may change where we host static data exports in the future. I would rather avoid adding another regex search but you're the Boss. Edit: Just for educational purpose here the regex expression I use to match the filename of all SDE releases so far. "._\d+\.\d+[\.\d+]_\d+[_db]\.zip\|sde-\d+-\w+[-legacy]\.zip" That's a gnarly regex. Why are you using Square brackets everywhere? Those only match single characters. It may be different based on languages but I believe you should change your [_db]* to (_db)? and your [-legacy]* to (-legacy)?
Desmont McCallock 604	Posted - 2016.04.26 17:32:43 - [42] - Quote No I don't, cause I don't want to group match. "_db" and "_legacy" may appear or not in the filename.
Cwittofur Cesaille Quantum Distributions Crit My Hole 4	Posted - 2016.04.26 17:45:01 - [43] - Quote Desmont McCallock wrote: No I don't, cause I don't want to group match. "_db" and "_legacy" may appear or not in the filename. That's what the ? is for, it makes it so it can appear but isn't required. I just read [_db]* as "zero or more of _ d or b"
Desmont McCallock 604	Posted - 2016.04.26 18:03:30 - [44] - Quote Cwittofur Cesaille wrote: Desmont McCallock wrote: No I don't, cause I don't want to group match. "_db" and "_legacy" may appear or not in the filename. That's what the ? is for, it makes it so it can appear but isn't required. I just read [_db]* as "zero or more of _ d or b" Parentheses are for grouping. I don't want that. Never the less, the regex expression works for me so...
Cwittofur Cesaille Quantum Distributions Crit My Hole 4	Posted - 2016.04.26 18:14:55 - [45] - Quote Desmont McCallock wrote: Cwittofur Cesaille wrote: Desmont McCallock wrote: No I don't, cause I don't want to group match. "_db" and "_legacy" may appear or not in the filename. That's what the ? is for, it makes it so it can appear but isn't required. I just read [_db]* as "zero or more of _ d or b" Parentheses are for grouping. I don't want that. Never the less, the regex expression works for me so... I know it's meant for grouping; I'm not being argumentative here I'm trying to understand. I was taught to use parenthesis when I needed a distinct pattern to appear. And adding the question mark at the end just makes the group optional. I don't see an issue with the grouping if you're not actively using it. Again; not trying to argue, I really just want to understand.
Desmont McCallock 604	Posted - 2016.04.26 18:19:44 - [46] - Quote OK. Here is some food for thought. Let's say we have the folowings: sde-20160426-TRANQUILITY.zip sde-20160426-TRANQUILITY-legacy.zip Parallax_1.0_115480_db.zip YC-118-3_1.0_117575.zip Aegis_1.1.1_114255_db.zip someotherfilename.zip Write a regex expression that matches with the first five (5) lines only without grouping.
Cwittofur Cesaille Quantum Distributions Crit My Hole 4	Posted - 2016.04.26 18:30:30 - [47] - Quote Point made. Why are you against grouping though?
Desmont McCallock 604	Posted - 2016.04.26 18:39:15 - [48] - Quote Cwittofur Cesaille wrote: Point made. Why are you against grouping though? I'm not. It just doesn't serve any purpose in this case.
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5978	Posted - 2016.04.26 19:50:07 - [49] - Quote Something to try: Rename a file to sde-20160426-TRANQUILITY-legcay.zip and see if it still matches. It may just be the regex engine I'm used to (PCRE), but I think it'll still match. I'm running again for CSM 11, and I'd appreciate your vote. Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.26 20:44:42 - [50] - Quote Desmont McCallock wrote: 3. Would you mind fixing the duplicate entries in blueprints.yaml file? (blueprintTypeID: 41590 has typeID: 38 for manufacturing specified two (2) times with different quantity) This his how it appears in-game: "Required Input Materials" for that blueprint lists 3x Nocxium and 5x Nocxium.

Desmont McCallock 604	Posted - 2016.04.26 21:37:29 - [51] - Quote Steve Ronuken wrote: Something to try: Rename a file to sde-20160426-TRANQUILITY-legcay.zip and see if it still matches. It may just be the regex engine I'm used to (PCRE), but I think it'll still match. It does Steve and in this case the only expression I could come up with to exclude that is ".*_\d+\.\d+(\.\d+)?_\d+(_db)?\.zip\|sde-\d+-\w+(-legacy)?\.zip" which justifies Cwittofur Cesaille case, but still there is no need to group matching. Never the less, this whole regex expression thing isn't about which is right but merely what suits needs.
Desmont McCallock 604	Posted - 2016.04.26 21:38:15 - [52] - Quote CCP Tellus wrote: Desmont McCallock wrote: 3. Would you mind fixing the duplicate entries in blueprints.yaml file? (blueprintTypeID: 41590 has typeID: 38 for manufacturing specified two (2) times with different quantity) This his how it appears in-game: "Required Input Materials" for that blueprint lists 3x Nocxium and 5x Nocxium. And from your experience is this correct? Should it be such?

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.26 22:16:58 - [53] - Quote Desmont McCallock wrote: And from your experience is this correct? Should it be such? That's not for me to judge. I suggest you file a bug report and one of the game design teams will take a look at it. :)

Desmont McCallock 604	Posted - 2016.04.27 09:47:28 - [54] - Quote @CCP Tellus Mate in legacy format, restoring the MSSQL datadump reveals that table dgmAtrributeTypes is empty.
Pete Butcher KarmaFleet Goonswarm Federation 324	Posted - 2016.04.27 13:16:45 - [55] - Quote And we're back to the integrity problems with the SDE. Someone should really clean it up. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor

CCP Tellus C C P C C P Alliance 59	Posted - 2016.04.27 14:26:42 - [56] - Quote Desmont McCallock wrote: Edit2: Importing the dgmAtrributeTypes table from the yaml file, I notice that a clean up has been made. 994 entries have been removed including some that we use in EVEMon data files generator. Is this intentional? That was not intentional. I'll make a new build of the SDE with several fixes, should have it ready in a few hours. Thanks for your patience. :)


CCP Tellus C C P C C P Alliance 60	Posted - 2016.04.27 21:11:54 - [57] - Quote Hello everyone! I published a new SDE that should hopefully contain fixes for all the reported issues. Please let us know if there are any more issues. https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY-legacy.zip

Shish Tukay Caldari Provisions Caldari State 38	Posted - 2016.04.27 21:51:30 - [58] - Quote Constructive: If you really hate the idea that you'll get more groups in output, you can also do "(?:_db)?" to make the group non-capturing while still being correct :) Criticism: Using square brackets really is incorrect though, not merely a matter of taste. Sure, it works, sort of, sometimes, if you ignore all the cases in which it doesn't work; but is it not better to work all the time? "(_db)?" = "may or may not contain '_db'", which is right; "[_db]" = "can contain any number of '_', 'd', or 'b'", eg it allows 'bbbbbbbbb_b_b_b___ddbbdbb', which is wrong. Still really confused why you would even want* to avoid groups though - that's like avoiding "else" statements and just writing "if(foo) {}; if(not foo) {}" all over the place because you don't like "if(foo) {} else {}" o___O
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5978	Posted - 2016.04.27 23:20:06 - [59] - Quote CCP Tellus wrote: Hello everyone! I published a new SDE that should hopefully contain fixes for all the reported issues. Please let us know if there are any more issues. https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY-legacy.zip \o/ <3 I'm running again for CSM 11, and I'd appreciate your vote. Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Desmont McCallock 604	Posted - 2016.04.28 08:07:40 - [60] - Quote Shish Tukay wrote: Constructive: If you really hate the idea that you'll get more groups in output, you can also do "(?:_db)?" to make the group non-capturing while still being correct :) Criticism: Using square brackets really is incorrect though, not merely a matter of taste. Sure, it works, sort of, sometimes, if you ignore all the cases in which it doesn't work; but is it not better to work all the time? "(_db)?" = "may or may not contain '_db'", which is right; "[_db]" = "can contain any number of '_', 'd', or 'b'", eg it allows 'bbbbbbbbb_b_b_b___ddbbdbb', which is wrong. Still really confused why you would even want* to avoid groups though - that's like avoiding "else" statements and just writing "if(foo) {}; if(not foo) {}" all over the place because you don't like "if(foo) {} else {}" o___O Thanks for your input. The matter is closed. Explicit capture option resolves any mismatch in my case.
Desmont McCallock 604	Posted - 2016.04.28 10:56:47 - [61] - Quote Bug report for data format inconsistency in invFlags table.
Hel O'Ween Men On A Mission 159	Posted - 2016.04.28 15:33:48 - [62] - Quote I've skimmed through the patch notes, but didn't find it mentioned somewhere, so I ask it here: 1) Which API provides the citadel's location? Would those be included in ConquerableStationList? 2) What "magic number(s)" - if any, is necessary for calculating the citadel's solar system from an asset's locationID? EVEWalletAware - an offline wallet manager.
Pete Butcher KarmaFleet Goonswarm Federation 326	Posted - 2016.04.28 15:42:43 - [63] - Quote Hel O'Ween wrote: I've skimmed through the patch notes, but didn't find it mentioned somewhere, so I ask it here: 1) Which API provides the citadel's location? Would those be included in ConquerableStationList? 2) What "magic number(s)" - if any, is necessary for calculating the citadel's solar system from an asset's locationID? https://forums.eveonline.com/default.aspx?g=posts&m=6456723#post6456723 http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Hel O'Ween Men On A Mission 159	Posted - 2016.04.28 16:55:38 - [64] - Quote Yeah, saw your thread, Pete, but as it hasn't got an answer, I thought I might menion it elsewhere (here) again. Or are you implying with your post that the locations are indeed in crest.industry.facilities, as Mr. mac suggested there? If so, that would totally break EWA ... :-( EVEWalletAware - an offline wallet manager.
Pete Butcher KarmaFleet Goonswarm Federation 326	Posted - 2016.04.28 17:02:06 - [65] - Quote Hel O'Ween wrote: Yeah, saw your thread, Pete, but as it hasn't got an answer, I thought I might menion it elsewhere (here) again. Or are you implying with your post that the locations are indeed in crest.industry.facilities, as Mr. mac suggested there? If so, that would totally break EWA ... :-( Nah, I didn't know you've seen that topic. I still have no idea how to handle citadels. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Desmont McCallock 604	Posted - 2016.04.29 08:13:57 - [66] - Quote @CCP Tellus region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however.

CCP Tellus C C P C C P Alliance 60	Posted - 2016.04.29 10:05:17 - [67] - Quote Desmont McCallock wrote: region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however. You'll need to calculate the radius. It should be the Euclidean distance between the region's 'min' and 'max', divided by the square root of two.

Desmont McCallock 604	Posted - 2016.04.29 10:35:29 - [68] - Quote CCP Tellus wrote: Desmont McCallock wrote: region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however. You'll need to calculate the radius. It should be the Euclidean distance between the region's 'min' and 'max', divided by the square root of two. Shouldn't the formula be radius = GêÜ((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ?

CCP Tellus C C P C C P Alliance 60	Posted - 2016.04.29 10:44:23 - [69] - Quote Desmont McCallock wrote: Shouldn't the formula be radius = GêÜ((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ? The distance between 'min' and 'max' is the diameter. You should divide that by two to get the radius, but someone messed it up by putting the division by two inside the square root for each field. The end result is that you divide the diameter with the square root of two instead. This is based on my understanding of quickly looking through the source that used to build the universe SQLite file. If the numbers don't check out, I'll take another stab at it for you. :)

Desmont McCallock 604	Posted - 2016.04.29 10:55:27 - [70] - Quote CCP Tellus wrote: Desmont McCallock wrote: Shouldn't the formula be radius = GêÜ((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ? The distance between 'min' and 'max' is the diameter. You should divide that by two to get the radius, but someone messed it up by putting the division by two inside the square root for each field. The end result is that you divide the diameter with the square root of two instead. This is based on my understanding of quickly looking through the source that used to build the universe SQLite file. If the numbers don't check out, I'll take another stab at it for you. :) OK. I assumed that a region is a sphere so I had to calculate the distance from the center, which apparently is incorrect. Fiddling around the formula becomes radius = (GêÜ((xMax - xMin)^2 + (yMax - yMin)^2 + (zMax - zMin)^2)) / 2. This gives me fairly the same numbers as the old universe format. Still need to figure out some rounding.
Desmont McCallock 604	Posted - 2016.04.29 13:17:48 - [71] - Quote @CCP Tellus I'm afraid that there is something fundamentally wrong with the formula you use to calculate the radius. According to known maths the radius of a sphere centered at the point (x0,y0,z0) derives from the following formula. r^2 = (x-x0)^2+(y-y0)^2+(z-z0)^2 which is equivalent to r = GêÜ((x-x0)^2+(y-y0)^2+(z-z0)^2) From my understanding 'min' coordinates are the coordinates of the celestial closest to the center and 'max' coordinates are the coordinates of the celestial farthest to the center. Using radius = (GêÜ((xMax - xMin)^2 + (yMax - yMin)^2 + (zMax - zMin)^2)) / 2 or even radius = (GêÜ(((xMax - xMin)^2 / 2) + ((yMax - yMin)^2 / 2) + ((zMax - zMin)^2 / 2))) / GêÜ2 (which is the formula you say you use) as the formula to calculate the radius seems wrong to me in two points. 1. The used formula is incorrect 2. The radius should be calculated from the center and not from the closest point to the center. Am I wrong in my assumptions?

CCP Tellus C C P C C P Alliance 60	Posted - 2016.04.29 15:33:43 - [72] - Quote Desmont McCallock wrote: I'm afraid that there is something fundamentally wrong with the formula you use to calculate the radius. Sorry for all the confusion. Took a stab at this and this is what I get: From the universe data SQLite file, we have that the radius is 5.57645447335415e+16. sqlite> select regionID, regionName, radius from mapRegions where regionName = 'A821-A'; regionID regionName radius ---------- ---------- -------------------- 10000019 A821-A 5.57645447335415e+16 By letting x1, y1, and z1 be the region's 'max' coordinates, and x2, y2, and z2 be the region's 'min' coordinates, we get: >>> math.sqrt((x1 - x2)2 + (y1 - y2)2 + (z1 - z2)**2) / 2 5.576454473354151e+16 This matches the radius of the region in the legacy SDE. It also turns out I was wrong about dividing by the square root of two. Terribly sorry. I should test code before reaching a conclusion. :( Hope this helps.

Desmont McCallock 606	Posted - 2016.04.29 15:48:50 - [73] - Quote My bad, it's indeed 5.57645447335415E+16 . My head was spinning and I posted the radius of Derelik. Will give it a try.
Masao Kurata Perkone Caldari State 485	Posted - 2016.04.29 15:56:08 - [74] - Quote That's a really strange definition of a "radius". That's half the diagonal of a cube.
Desmont McCallock 606	Posted - 2016.04.29 15:56:26 - [75] - Quote Still I believe the formula is wrong. math.sqrt((x1 - x2)2 + (y1 - y2)2 + (z1 - z2)**2) IS the Euclidean distance which IS the radius for a sphere. It shouldn't be divided by two and x2,y2,z2 should be the center coordinates and not the min ones. Someone confused distance with diameter. Never the less, even if it's wrong I guess it won't be fixed as it's been set to stones for years, right?

CCP Tellus C C P C C P Alliance 60	Posted - 2016.04.29 16:22:01 - [76] - Quote Desmont McCallock wrote: Never the less, even if it's wrong I guess it won't be fixed as it's been set to stone for years in the SDE, right? Pretty much.

Desmont McCallock 606	Posted - 2016.04.29 16:56:13 - [77] - Quote @ CCP Tellus I can't match the numbers that the SDE had for the region radius. Probably because of the coordinates rounding or .NET has different implementation for math.sqrt than Python. Is it to much to ask to include the region radius info in the universe yaml file (region.staticdata)? It doesn't have to be this one. The next one maybe?
Desmont McCallock 606	Posted - 2016.05.01 14:19:42 - [78] - Quote After 5 long days battling with the new SDE format I'm finally done. CCP Tellus I know I have become a pain in your bud but you had it coming (joking). Here are my findings: Errors - provide 'radius' info in region.staticdata for each region (already been discussed in this thread why it should be provided) - landmarks are missing landmarkName, description (instead their ID counterparts are provided) - constellation 'Tranquility' in Curse is named 'Tranquillity' (notice the double 'l') - constellation 'G-C00324' in G-R00031 is named 'G-C00311' (which is correct?) - solar system 'J170376' in 'C-R00015 > C-C00146' is named 'J140208' (which is correct?) - provide 'orbitIndex' info in order to be able to produce the correct itemName for mapDenormalize (mainly for asteroid belts and moons) - provide the special name of a celestial (sun, planet, moon etc) if exists (i.e. Uplingur IV (Ndoria), New Caldari I (Matigu), New Caldari III (Orieku), New Caldari V (Oniteseru)) - provide the npcStations name in order to be able to produce the correct itemName for mapDenormalize (please don't tell me that it can be found in staStations, legacy sde provided that without needing statStations) Positive points - Landmarks iconID can be now placed in the correct column (in legacy SDE it's falsely placed in radius column) - mapCelestialStatistics 'pressure' info are more precise (it's truncated in legacy) - A lot more data is available in new SDE format (but I'll stick on providing what was in legacy for EVESDEToSQL, at least for now) Questions - In legacy mapDenormilize table stargates and NPCStations have a number in radius. Where do we find that in the new SDE format?
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5984	Posted - 2016.05.01 18:12:13 - [79] - Quote All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations. Radius and orbit index are, indeed, a bit of a pain. Though both can be calculated. (I need to work in the radius calculation) Landmark descriptions are in trnTranslations now. (side effect of how the files are generated.) The stargate Radius, I suspect, comes from typeIDs.yaml I'm running again for CSM 11, and I'd appreciate your vote. Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Desmont McCallock 607	Posted - 2016.05.01 18:27:43 - [80] - Quote Steve Ronuken wrote: All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations. Steve, sorry man but you don't get it. Universe data need to be independent as they always were. Use case? If someone wants to import only the universe data back to a DB, he ends up with missing data. Steve Ronuken wrote: Radius and orbit index are, indeed, a bit of a pain. Though both can be calculated. (I need to work in the radius calculation) About region radius we already discussed that in length with CCP Tellus in this thread (read previous posts). Numbers only match if you use Python. As for orbitIndex, it's because I tried to calculate them, I'm asking about that info to be included. The index order isn't the same for all celestials and it gets all messed up. Steve Ronuken wrote: Landmark descriptions are in trnTranslations now. (side effect of how the files are generated.) Same case as with celestial names. Steve Ronuken wrote: The stargate Radius, I suspect, comes from typeIDs.yaml Still same case as celestial names. Independency.
Carbon Alabel The Alabaster Albatross Sev3rance 11	Posted - 2016.05.01 18:50:14 - [81] - Quote Desmont McCallock wrote: Universe data need to be independent as they always were. As far as I know, the SDE was always distributed in the form of a single archive file, so how exactly was it independent?
Desmont McCallock 607	Posted - 2016.05.01 19:40:52 - [82] - Quote Carbon Alabel wrote: Desmont McCallock wrote: Universe data need to be independent as they always were. As far as I know, the SDE was always distributed in the form of a single archive file, so how exactly was it independent? In the form that you didn't need to do a lookup on another table. itemName in mapDenormalize was always provided. You didn't had to do a lookup on another table to get the data.
Desmont McCallock 609	Posted - 2016.05.03 10:02:14 - [83] - Quote Something interesting I found while working with re-importing the new SDE files back into a DB is that the 'mapSolarSystemJumps', 'mapConstellationJumps', 'mapRegionJumps' tables can be easily created with a query on 'mapJumps' and 'mapDenormalize' tables. You may say "Pfff, no big deal...", but here they are for future reference. /* mapSolarSystemJumps / SELECT denorm1.regionID as fromRegionID, denorm1.constellationID as fromConstellationID, denorm1.solarSystemID as fromSolarSystemID, denorm2.solarSystemID as toSolarSystemID, denorm2.constellationID as toConstellationID, denorm2.regionID as toRegionID FROM mapJumps as jumps JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID ORDER BY denorm1.solarSystemID, denorm2.solarSystemID / mapConstellationJumps / SELECT denorm1.regionID as fromRegionID, denorm1.constellationID as fromConstellationID, denorm2.constellationID as toConstellationID, denorm2.regionID as toRegionID FROM mapJumps as jumps JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID WHERE denorm1.constellationID != denorm2.constellationID GROUP BY denorm1.regionID, denorm1.constellationID, denorm2.constellationID, denorm2.regionID ORDER BY denorm1.constellationID, denorm2.constellationID / mapRegionJumps */ SELECT denorm1.regionID as fromRegionID, denorm2.regionID as toRegionID FROM mapJumps as jumps JOIN mapDenormalize as denorm1 on jumps.stargateID = denorm1.itemID JOIN mapDenormalize as denorm2 on jumps.celestialID = denorm2.itemID WHERE denorm1.regionID != denorm2.regionID GROUP BY denorm1.regionID, denorm2.regionID ORDER BY denorm1.regionID, denorm2.regionID
Desmont McCallock 609	Posted - 2016.05.06 14:19:22 - [84] - Quote Steve Ronuken wrote: All names are in invNames, so you can look them up, and insert them. This is for asteroid belts, moons, planets, stars, stations. Just out of curiosity I wanted to give this a try, so I did some checking via select invNames.itemID, invNames.itemName as InvNames_itemName, mapDenormalize.itemName as MapDenormalize_itemName from dbo.invNames as invNames join dbo.mapDenormalize as mapDenormalize on invNames.itemID = mapDenormalize.itemID where invNames.itemName <> mapDenormalize.itemName The result comes back with 122 rows, proving another data inconsistency in SDE. This is getting nowhere.
Desmont McCallock 614	Posted - 2016.05.08 09:14:07 - [85] - Quote Yet another data inconsistency with names. select invNames.itemID, invNames.itemName as InvNames_itemName, invUniqueNames.itemName as InvUniqueNames_itemName from dbo.invNames as invNames join dbo.invUniqueNames as invUniqueNames on invNames.itemID = invUniqueNames.itemID where invNames.itemName <> invUniqueNames.itemName collate Latin1_General_BIN This time the query returns 6994 rows.
Hurt Kion Hurt Kion's Inc. 0	Posted - 2016.05.14 23:00:03 - [86] - Quote Some blueprints materials in SDK (blueprints.yaml normal and legacy) do not match Tranquility. For example: CONCORD Capital Shield Extender Blueprint in SDK: 41603: activities: manufacturing: materials: - quantity: 1 typeID: 40354 [ Capital Shield Extender I ] - quantity: 8 typeID: 41267 [ Compact Compounds ] - quantity: 12 typeID: 41308 [ Restrained Compounds ] - quantity: 41 typeID: 41268 [ Compact Conductors ] - quantity: 44 typeID: 41266 [ Compact Electronics ] - quantity: 47 typeID: 41309 [ Restrained Conductors ] - quantity: 50 typeID: 41307 [ Restrained Electronics ] products: - quantity: 1 typeID: 41459 skills: - level: 1 typeID: 3380 - level: 1 typeID: 22242 time: 36000 blueprintTypeID: 41603 maxProductionLimit: 5 in game: 1 x Capital Shield Extender II 13 x Broadcast Node 20 x Sterile Conduits 10 x Wetware Mainframe Same with: CONCORD 25000mm Steel Plates Blueprint CONCORD Capital Armor Repairer Blueprint maybe others...
Jai Blaze Honor Forge Joint Operation Involving Nobodys 0	Posted - 2016.05.17 04:21:48 - [87] - Quote Please more JSON files!! I've tried converting typeIDs.yaml into JSON and it refuses to validate. It's the single most important file for my own use, blueprints converted nicely and a lot of the others are small enough I can excel the crap out of it and reform it into a JSON file manually, but typeIDs is too big and when I try to convert manually it crashes. And nothing I've found so far will accept it and return a JSON file. MOAR JSON! please :)
Myme Temet'Nosce New Horizon Entreprise Le Consortium Horizon 0	Posted - 2016.05.22 12:51:36 - [88] - Quote Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ? What did I miss ?
Pete Butcher KarmaFleet Goonswarm Federation 332	Posted - 2016.05.22 13:09:40 - [89] - Quote Myme Temet'Nosce wrote: Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ? What did I miss ? Sql is neither standardized, very fast or efficient. Just trowing my 2 cents there. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5995	Posted - 2016.05.22 14:50:55 - [90] - Quote Myme Temet'Nosce wrote: Can someone explain clearly why getting rid of a standardized, very fast and very efficient format as sql to use yaml, which is not as good by far ? What did I miss ? Try version controlling a database. Then you'll realise why. Total PITA. And what works in one database, doesn't always work in another. (And Eve uses at least three. SQL Server on the server, sqlite and a custom format on the client.) What we're getting now, is a the output from their version control system, rather than some multi step process, which is a pain in the ass for them to manage, and a pain in the ass for people to convert to other database formats. For me, having it all moved into yaml is a godsend, as I can write a single script, which takes the yaml, and dumps in to whatever database I want it to. Rather than the multi step process which I needed to run through before. It's significantly easier, requiring a lot less attention. Woo! CSM XI! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5995	Posted - 2016.05.22 15:21:04 - [91] - Quote Jai Blaze wrote: Please more JSON files!! I've tried converting typeIDs.yaml into JSON and it refuses to validate. It's the single most important file for my own use, blueprints converted nicely and a lot of the others are small enough I can excel the crap out of it and reform it into a JSON file manually, but typeIDs is too big and when I try to convert manually it crashes. And nothing I've found so far will accept it and return a JSON file. MOAR JSON! please :) https://www.fuzzwork.co.uk/dump/sde-20160429-TRANQUILITY/typeid.json.bz2 ? Converted with: https://github.com/fuzzysteve/yamlloader/blob/master/TypesToJson.py Works in python 2.7 (it'd work in 3.5, except I'm not doing the prints right for that) needs pyyaml, with the libyaml library installed. Change the loader to loader, rather than cloader to work (slower) without libyaml. May need to be 64 bit, for memory usage on parsing. Woo! CSM XI! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Mathliney Kagrel Too Many Miners 0	Posted - 2016.05.22 15:27:58 - [92] - Quote Steve Ronuken wrote: Try version controlling a database. Then you'll realise why. Total PITA. And what works in one database, doesn't always work in another. (And Eve uses at least three. SQL Server on the server, sqlite and a custom format on the client.) Ok, versioning is not easy but is it really required ? Quote: What we're getting now, is a the output from their version control system, rather than some multi step process, which is a pain in the ass for them to manage, and a pain in the ass for people to convert to other database formats. I can understand the need to unify but why chosing YAML ? What is the advantage beside versioning, which is not really important imho ? Quote: For me, having it all moved into yaml is a godsend, as I can write a single script, which takes the yaml, and dumps in to whatever database I want it to. Rather than the multi step process which I needed to run through before. It's significantly easier, requiring a lot less attention. Yes, thanx to you about this btw. Now what we are left with is waiting for you to export SDE in SQL format or build our own tool to convert data. Pete Butcher wrote: Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there. Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ?
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5995	Posted - 2016.05.22 15:36:28 - [93] - Quote Versioning isn't important. Really? Are you really trying to say that? Are you kidding me? Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important. As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released. As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names? Woo! CSM XI! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Pete Butcher KarmaFleet Goonswarm Federation 333	Posted - 2016.05.22 15:37:42 - [94] - Quote Mathliney Kagrel wrote: Pete Butcher wrote: Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there. Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ? Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Mathliney Kagrel Too Many Miners 0	Posted - 2016.05.22 16:09:03 - [95] - Quote Pete Butcher wrote: Mathliney Kagrel wrote: Pete Butcher wrote: Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there. Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ? Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized. The fact that your MySQL database doesn't respect standards doesn't imply SQL isn't standardized.
Mathliney Kagrel Too Many Miners 0	Posted - 2016.05.22 16:14:09 - [96] - Quote Steve Ronuken wrote: Versioning isn't important. Really? Are you really trying to say that? Are you kidding me? Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important. As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released. As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names? It is an export, I never implied CCP should work with SQL and not YAML, we are talking about SDE here. I know about your script, I don't use python but could give a try, I can also create my own tool, this is not the point. My point was : why not using something that is more than proven for ages and with very rich frameworks to handle it, no matter the language / IDE you are using (not to mention the Entity Framework for example) to use YAML ? I am just asking, I didn't say it was a bad choice, I'm just curious, I don't think the majority of ppl creating tools for EVE are using YAML for something else that config files or tiny serialization needs. Now, maybe I'm wrong. And yes, of course I follow SQL standards, are they that big constraint ? Won't reply further on this topic, I have my answer as you stated something I read before : it is easier and more convenient for them and since some tools exist to update our own products, I'll deal with it. Thank you for your replies (both of you).
Pete Butcher KarmaFleet Goonswarm Federation 333	Posted - 2016.05.22 16:19:08 - [97] - Quote Mathliney Kagrel wrote: Pete Butcher wrote: Mathliney Kagrel wrote: Pete Butcher wrote: Sql is neither standardized, very fast nor efficient. Just trowing my 2 cents there. Check wiki about standard, it is clearly. About efficiency and speed, I hope you are not comparing with YAML, are you ? Yeah, the wiki... Run SELECT TOP 1 in MySQL and, please, tell me again how it's standardized. The fact that your MySQL database doesn't respect standards doesn't imply SQL isn't standardized. Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun. To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 5997	Posted - 2016.05.22 16:35:35 - [98] - Quote Mathliney Kagrel wrote: Steve Ronuken wrote: Versioning isn't important. Really? Are you really trying to say that? Are you kidding me? Yes, it might not be as important for us, as consumers. But for CCP, when they have multiple streams, which need to be merged together, you'd better believe it's important. As for 'waiting for me to export it' : https://github.com/fuzzysteve/yamlloader go knock yourself out. It's been available since a couple of days after the SDE was properly released. As for SQL being standardized, yes, there are SQL standards. Which does your database conform to? Does it implement all of it? (no, it doesn't.) Do you use double quotes round identifiers? back ticks? Are identifiers case sensitive? Do you put square brackets round your table names? It is an export, I never implied CCP should work with SQL and not YAML, we are talking about SDE here. I know about your script, I don't use python but could give a try, I can also create my own tool, this is not the point. My point was : why not using something that is more than proven for ages and with very rich frameworks to handle it, no matter the language / IDE you are using (not to mention the Entity Framework for example) to use YAML ? I am just asking, I didn't say it was a bad choice, I'm just curious, I don't think the majority of ppl creating tools for EVE are using YAML for something else that config files or tiny serialization needs. Now, maybe I'm wrong. And yes, of course I follow SQL standards, are they that big constraint ? Won't reply further on this topic, I have my answer as you stated something I read before : it is easier and more convenient for them and since some tools exist to update our own products, I'll deal with it. Thank you for your replies (both of you). tbh, yaml isn't being used as a final data source in this case. It's being used to transfer data. Which works well. It's easy to convert from yaml to something else. And that's the real benefit. If they were releasing sql server database backup (which was how it used to be) it was a pain to migrate, requiring some manual steps. And very few people other than CCP were using it. And anything else would increase their costs to release the SDE. This way, they can automate it, and hopefully get more releases done. including, for example, exports from SiSi. It's also far less likely to break (which happened sometimes, leading to a delay in getting it out.) Yaml is far from perfect, but it fits this use case well. Woo! CSM XI! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Mathliney Kagrel Too Many Miners 0	Posted - 2016.05.22 17:48:57 - [99] - Quote Pete Butcher wrote: Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun. To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses. Once again, you are out of topic here : there IS a SQL standard, you are only stating that this standard isn't enforced / respected by every engine. Great, there is a standard nonetheless. Now, could you calm down your need to feel superior and speak with others like they are kids / noobs ? You have no clue who I am nor in what domain I'm an expert, you don't know what experience I have with databse, you only use a side case to make your point. This is not the first time you reply with this tone, you have a very high opinion of yourself, we don't all share it.
Pete Butcher KarmaFleet Goonswarm Federation 333	Posted - 2016.05.22 17:58:42 - [100] - Quote Mathliney Kagrel wrote: Pete Butcher wrote: Then run SELECT ... LIMIT 1 on Oracle or SELECT ... WHERE ROWNUM < 2 on SQL Server. Or use [] in MySQL. Or the same case-insensitive query in Oracle as In MySql. Have fun. To put this in short - SQL is very poorly standardized. So poorly that you can say each RDBMS has its own. If you want to dispute this further, take some time and do research first, or get some experience with different databses. Once again, you are out of topic here : there IS a SQL standard, you are only stating that this standard isn't enforced / respected by every engine. Great, there is a standard nonetheless. Now, could you calm down your need to feel superior and speak with others like they are kids / noobs ? You have no clue who I am nor in what domain I'm an expert, you don't know what experience I have with databse, you only use a side case to make your point. This is not the first time you reply with this tone, you have a very high opinion of yourself, we don't all share it. Non-enforced standard isn't a standard. It's one of the standards. I don't wish to sound superior, but I don't want misinformation to spread. You made three claims, all of which are false. http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
Golden Gnu EVE University Ivy League 182	Posted - 2016.05.24 22:41:13 - [101] - Quote For some reason some of the station names in staStations.yaml doesn't match up with that is in invNames.yaml (I assume the once in staStations.yaml is the correct ones, as that is the names used in the legacy SDE). invNames.itemID: from 60014861 to 60014931 Are missing "Serpentis" or worse (see examples). Ex 1.: staStations: Polaris VIII - Serpentis Corporation Cloning invNames: Polaris VIII - Cloning Outpost Ex 2.: staStations: NOL-M9 VI - Moon 2 - Serpentis Corporation Cloning invNames: NOL-M9 VI - Moon 2 - NOL Thank you for releasing the data in yaml. It was a fun task writing a new backend to generate data for jEveAssets Ofc. I would have just continued to use Desmont McCallock's tool if he hadn't quite Creator of jEveAssets - the asset manager "Download is the meaning of life, upload is the meaning of intelligent life"
Mr Mac Dark Goliath 139	Posted - 2016.05.25 08:15:00 - [102] - Quote Those are outposts. You better pull real name from crest
Golden Gnu EVE University Ivy League 182	Posted - 2016.05.25 09:47:07 - [103] - Quote @Mr Mac Thank you. I just spotted the differences in the data, didn't think to do any research When you say crest, what end point are you talking about? I'm already doing eve/ConquerableStationList.xml.aspx from the XML API ...Do I need to get more than that? Creator of jEveAssets - the asset manager "Download is the meaning of life, upload is the meaning of intelligent life"
Mr Mac Dark Goliath 139	Posted - 2016.05.25 10:32:33 - [104] - Quote Golden Gnu wrote: @Mr Mac Thank you. I just spotted the differences in the data, didn't think to do any research When you say crest, what end point are you talking about? I'm already doing eve/ConquerableStationList.xml.aspx from the XML API ...Do I need to get more than that? /industry/facilities
Jai Blaze Honor Forge Joint Operation Involving Nobodys 0	Posted - 2016.05.27 19:28:40 - [105] - Quote Steve Ronuken wrote: Jai Blaze wrote: Stuff https://www.fuzzwork.co.uk/dump/sde-20160429-TRANQUILITY/typeid.json.bz2 ? Converted with: https://github.com/fuzzysteve/yamlloader/blob/master/TypesToJson.py Works in python 2.7 (it'd work in 3.5, except I'm not doing the prints right for that) needs pyyaml, with the libyaml library installed. Change the loader to loader, rather than cloader to work (slower) without libyaml. May need to be 64 bit, for memory usage on parsing. Ahh, I haven't dabbled in python yet, I'm actually currently working on a nodejs version of my industry spreadsheet. I ended up making my own typeIDs.json through a painful excel process haha. I found my excel industry sheet too difficult to maintain and update, and updating info through nodejs using crest is sooo fast, and I've got functions prepped to update my entire database using fresh json files or crest pulls depending on the data. Still new to nodejs. I'm only half way through a udemy course on nodejs and I started to work on something eve-related as a way to give myself an incentive to keep at it.
nezroy Nice Clan 6	Posted - 2016.05.28 01:13:35 - [106] - Quote Jai Blaze wrote: Ahh, I haven't dabbled in python yet, I'm actually currently working on a nodejs version of my industry spreadsheet. I ended up making my own typeIDs.json through a painful excel process haha. I found my excel industry sheet too difficult to maintain and update, and updating info through nodejs using crest is sooo fast, and I've got functions prepped to update my entire database using fresh json files or crest pulls depending on the data. Still new to nodejs. I'm only half way through a udemy course on nodejs and I started to work on something eve-related as a way to give myself an incentive to keep at it. Check out the EVEoj NPM package (and related site with data files). It has the SDE data you need ready to go for Node.js (or browser too).

Pages: 1 2 3 4 :: [one page]

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.