Pages: [1] 2 3 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 17 post(s) |
|
CCP Tellus
C C P C C P Alliance
36
|
Posted - 2016.02.10 18:24:23 -
[1] - Quote
Hello everyone,
We know that some of you have been frustrated with late SDE releases and last-minute-changes in the past. Building and deploying the SDE has always been a manual process that had a tendency to slip through the cracks.
We want to build and ship the SDE to you all sooner, and more often alongside both Tranquility and Singularity releases. This would necessitate some changes to the SDE and its build process in order to achieve continuous delivery.
In the process of migrating static data from our legacy BSD content authoring tool to our new FSD authoring tool; we've been moving away from editing tables in a centralized SQL Server database, to YAML files tracked in revision control. This process is reflected by the SDE, having shipped both a SQL Server backup file and YAML files for some time now. We don't anticipate this migration to finish anytime soon.
One way of simplifying the build process can be achieved by naïvely converting the SQL tables to YAML files.
As an example, the last two entries of the eveUnits.yaml file would look like:
- unitID: 141 unitName: "Hardpoints" displayName: "hardpoints" description: "For various counts to do with turret, launcher and rig hardpoints" - unitID: 142 unitName: Sex displayName: "1=Male 2=Unisex 3=Female" description: null
The first entry of staStations.yaml would look like:
- stationID: 60000004 security: 0 dockingCostPerVolume: 0.0 maxShipVolumeDockable: 50000000.0 officeRentalCost: 10000 operationID: 26 stationTypeID: 1531 corporationID: 1000002 solarSystemID: 30002780 constellationID: 20000407 regionID: 10000033 stationName: "Muvolailen X - Moon 3 - CBD Corporation Storage" x: 1723680890880.0 y: 256414064640.0 z: -60755435520.0 reprocessingEfficiency: 0.5 reprocessingStationsTake: 0.05 reprocessingHangarFlag: 4
In order to smooth out such a transition, we would of course ship the good o' SQL Server backup file and the SQLite universe data alongside these new YAML files for the next one or two SDE releases for Tranquility. That way you'd have time to update any tools you may be running on top of the Static Data Export.
What are your thoughts and opinions on this proposal?
Thanks! |
|
Desmont McCallock
598
|
Posted - 2016.02.10 19:48:55 -
[2] - Quote
tl;dr Get done with the transition and give as one format and make it an MySQL dump one.
Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats. I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE. And most like this format would be an MySQL dump.
From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them. |
Florian Lousberg
Furtherance.
2
|
Posted - 2016.02.10 20:22:21 -
[3] - Quote
I personally would welcome getting all the SDE tables in plain YAML on a more regular basis from an official source. For me it would be much easier to handle than waiting for FuzzySteves MySQL conversion (thanks for that, Steve! You got my vote!) and then converting that again into the custom Django database schemes I use in my projects.
Besides, the ability to just run a plain text diff against the previous version would be really nice, too. |
Zifrian
Licentia Ex Vereor Phoebe Freeport Republic
1707
|
Posted - 2016.02.10 21:01:54 -
[4] - Quote
As much as I dislike yaml, I'd rather you just publish the entire SDE in yaml for a few cycles along with the SQL server file and then phase the SQL Server file out for good. Because what I'm going to do is convert the data into what I want to use and then take only the data I need for my app. It would be great if you provided a single DB format (SQLite like the universe data?) that we could use but I don't care as long it's all the same. If yaml works for you best, then do it in that.
If I read your post correctly, it sounds like you want to add just two more files to yaml? Many of us have been dealing with this hodgepodge system for 4 years now and I'd just like you to get on with it and do it instead of this piecemeal approach. If you are going to release a few files here and there, then that is 100xs more frustrating to me than the current system and I'd rather you do nothing instead.
GÇ£Any fool can criticize, condemn, and complain - and most fools do. GÇ¥ - Dale Carnegie
Maximze your Industry Potential! - Download EVE Isk per Hour!
|
|
CCP Tellus
C C P C C P Alliance
37
|
Posted - 2016.02.10 21:10:02 -
[5] - Quote
Zifrian wrote:If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.
One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.
There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. |
|
Zifrian
Licentia Ex Vereor Phoebe Freeport Republic
1707
|
Posted - 2016.02.10 22:05:28 -
[6] - Quote
CCP Tellus wrote:Zifrian wrote:If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files. One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour. There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals. OK that's great then. I'm glad it's going to be all at once. Many of us have wanted this for years and totally understand your reasons for doing it.
Will the universe data also be migrated from SQLite? Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live?
Thanks for the update, looking forward to this.
GÇ£Any fool can criticize, condemn, and complain - and most fools do. GÇ¥ - Dale Carnegie
Maximze your Industry Potential! - Download EVE Isk per Hour!
|
|
CCP Tellus
C C P C C P Alliance
38
|
Posted - 2016.02.10 22:22:30 -
[7] - Quote
Zifrian wrote:Will the universe data also be migrated from SQLite? The universe data is authored as YAML files internally. These are then post-processed into an SQLite database. Why? I am not sure exactly. I'm contemplating just removing that post-processing step.
Zifrian wrote:Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live? That's the idea! Automatically building SDE exports for Singularity as soon as changes are deployed there shouldn't be much trouble either with these changes. |
|
Max Kolonko
WATAHA. Fidelas Constans
588
|
Posted - 2016.02.10 22:51:17 -
[8] - Quote
As long as some wizard like steve will provide his service of converting them to something old dog like me can understand I'm fine :)
Read and support:
Don't mess with OUR WH's
What is Your stance on WH stuff?
|
Dragonaire
Here there be Dragons
78
|
Posted - 2016.02.11 00:20:55 -
[9] - Quote
Not that I deal with the SDE directly to much but having everything in a single format should make dealing with it easier IMHO. Yaml is a good format for doing other conversions, from what I've seen and depending on the app it might be usable directly too. I would think Steve's conversion tool would also be easier to maintain and automate with just the one format.
Finds camping stations from the inside much easier.
Designer of Yapeal for the Eve API.
Check out the Yapeal PHP API Library thread.
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
5875
|
Posted - 2016.02.11 00:57:13 -
[10] - Quote
Go for it.
My conversion will be a lot simpler.
And easily transferable to other people. (i.e., if you have a fairly basic python install, you'll be able to import it. Into any database that sqlalchemy supports.)
Granted, it's going to be one big import file (probably) and it'll take a while to import, but it's a one off job. And I can do it directly on the server.
Woo! CSM X!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
|
Masao Kurata
Perkone Caldari State
350
|
Posted - 2016.02.11 01:03:45 -
[11] - Quote
YAML as the one true format sounds great, doing conversions yourself is too painful otherwise. |
Pete Butcher
KarmaFleet Goonswarm Federation
310
|
Posted - 2016.02.11 05:28:20 -
[12] - Quote
Do you plan to convert mapDenormalize too? That would be one big ass YAML
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Mr Mac
Dark Goliath
124
|
Posted - 2016.02.11 06:46:33 -
[13] - Quote
I like less YAML but having entire SDE in single format is fine. |
Desmont McCallock
598
|
Posted - 2016.02.11 08:22:19 -
[14] - Quote
I agree with Steve. Our tools would have more meaning (to the linux savvies out there) if we could get rid of the SQL Server. In that case I could even make my tool target .NET Core and have it running on every OS.
On another note, as Pete already mentioned, what about the mapDenormalize table? This will produce the largest YAML file. Do we have a size number so we know approximately how big it's going to be? |
Evanova Android
Traquenard Labs
25
|
Posted - 2016.02.11 13:13:05 -
[15] - Quote
I can only speak for myself but so far, the YAML files have been an inconvenience more than anything; a pre-built SQL database is much easier for me to manipulate and use. I would guess that many applications will want to build a database out of the YAML files anyway, just to be able to create complex views out of many files/tables.
Also, having to deal with it on a daily basis, I also dislike YAML very much and I think it is The Wrong Format for pretty much everything, but that's a personal and not very popular opinion.
Now, in all practical sense, if exporting YAML files makes the data set more consistent, easier to update and simple to use and build things out of for CCP and datameister Fuzzy, that cannot be a bad change.
Evanova - The Android App for Eve Online
|
BigSako
Burning Napalm Northern Coalition.
120
|
Posted - 2016.02.11 13:17:19 -
[16] - Quote
Why do we need yet another format? The SDE should be in the same format as CREST, and that is JSON.
JSON is easy to parse and you can directly use it in MANY applications, making (manual) converting unnecessary!
edit: I would also like to see an index of the SDE, showing two things per "table" (file): Last Update + Update History / Diff |
Pete Butcher
KarmaFleet Goonswarm Federation
311
|
Posted - 2016.02.11 14:13:35 -
[17] - Quote
While we're at it, it might be a good idea to finally fix integrity problems within the SDE itself. There are relations to non-existing entities.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Shala Raan
Trillionaire.pro
28
|
Posted - 2016.02.11 14:41:38 -
[18] - Quote
This looks nice. I use Steve's converions so if it helps him I'm all in
(püñGÿ»ßù£Gÿ»)püñ TRILLIONAIRE.PRO
|
Erick1111
Quarantine Corporation
0
|
Posted - 2016.02.11 18:43:22 -
[19] - Quote
Having a one and only data format would certainly be a blessing. Then if we have conversions to make, we can make them internally. |
Matt Faithbringer
Ravens Of Faith and Light
29
|
Posted - 2016.02.16 16:18:05 -
[20] - Quote
Desmont McCallock wrote:tl;dr Get done with the transition and give as one format and make it an MySQL dump one.
Since 2012, when the first YAML file appeared we are dealing with the splitting of the SDE to three (3) formats. I'm with the impression, and correct me if I'm wrong, a developer would like to work with one format when it comes down to SDE. And most like this format would be an MySQL dump.
From my perspective, I won't see anyone use the YAML and SQLite format in their projects, not until every table of the SQL database becomes a YAML or SQLite file. And even in that case, I bet all my ISKs on that, we will develop a tool to import those files either into the SQLite file, in order to have one format, or build a MySQL dump out of them.
I don't agree here. Not everyone uses MySQL. Some people use sqlite, some people use postgres. IMO it's bad idea to go with one format. Just migrate the whole thing to YAML, everyone can read that. |
|
Matt Faithbringer
Kill'em all. Let Bob sort'em out.
29
|
Posted - 2016.02.16 16:19:17 -
[21] - Quote
CCP Tellus wrote:Zifrian wrote:If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files. One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour. There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. This may not result in the prettiest YAML files, at least not until we migrate more data over to FSD internally. This approach is, in my opinion at least, the simplest and least intrusive method of achieving the stated goals.
I'm not big fan of YAML, but having everything in one format would be awesome, even when that format is YAML. |
Golden Gnu
EVE University Ivy League
179
|
Posted - 2016.02.22 15:46:15 -
[22] - Quote
For me it doesn't matter directly. I will probably always use the conversion and/or tools by Desmont McCallock or Steve Ronuken. However, if you make their job easier, you're doing a good job.
tl:dr I support whatever Desmont McCallock and Steve Ronuken says
Creator of jEveAssets - the asset manager
"Download is the meaning of life, upload is the meaning of intelligent life"
|
Mr Mac
Dark Goliath
137
|
Posted - 2016.04.25 15:34:44 -
[23] - Quote
Patch notes 2016-04-25
Quote:Static Data Export (SDE):
The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files. |
Desmont McCallock
604
|
Posted - 2016.04.25 15:39:33 -
[24] - Quote
Mr Mac wrote:Patch notes 2016-04-25Quote:Static Data Export (SDE):
The universe data and the SQL Server Backup file have been converted to YAML. The SDE now consists solely of YAML files. This is HUGE news. We need a pre-release asap. |
Pete Butcher
KarmaFleet Goonswarm Federation
324
|
Posted - 2016.04.25 18:20:51 -
[25] - Quote
mapDenormalize too? Holy crap, this must be a 1GB of yaml.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
5978
|
Posted - 2016.04.25 19:19:41 -
[26] - Quote
https://cdn1.eveonline.com/data/sde/singularity/sde-20160416-2016-DANIELLE.zip is the prerelease version which has been floating round Slack. (be aware there's a couple of dodgy numbers in the blueprint data. it's not an export problem, just a problem with the source, which should now be fixed)
I've got everything but the universe data loaded easily. Universe data shouldn't be too hard, but it'll need a chunk of processing, to make it look like the old data.
(Fun story: mapDenormalize isn't how they store it at CCP. This is a cut from their versioning system.)
Biggest differences so far:
All the tables which were in the sql server export are now in single yaml files. loading them is just a case of iterating through the array and loading them back into a table. some tables are a touch troublesome due to the number of entries (invNames, for example), but most load just fine. ( https://github.com/fuzzysteve/yamlloader )
I'm running again for CSM 11, and I'd appreciate your vote.
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Desmont McCallock
604
|
Posted - 2016.04.25 21:36:22 -
[27] - Quote
Judging from Steve's linked SDE, universe data is split down to each individual solarsystem, which makes mapDernomalize table creation a process 3rd party tools have to undertake.
BTW, thank you Steve for the link. |
|
CCP FoxFour
C C P C C P Alliance
4291
|
Posted - 2016.04.25 21:48:44 -
[28] - Quote
Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit.
@CCP_FoxFour // Technical Designer // Team Tech Co
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Desmont McCallock
604
|
Posted - 2016.04.25 22:11:43 -
[29] - Quote
CCP FoxFour wrote:Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit. Feeewww. You got me worried there for some time. |
|
CCP Tellus
C C P C C P Alliance
57
|
Posted - 2016.04.26 13:07:47 -
[30] - Quote
There's a new SDE release available on our developes' site:
https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY-legacy.zip |
|
|
|
|
|
Pages: [1] 2 3 4 :: one page |
First page | Previous page | Next page | Last page |