|
Author |
Thread Statistics | Show CCP posts - 17 post(s) |
|
CCP Tellus
C C P C C P Alliance
36
|
Posted - 2016.02.10 18:24:23 -
[1] - Quote
Hello everyone,
We know that some of you have been frustrated with late SDE releases and last-minute-changes in the past. Building and deploying the SDE has always been a manual process that had a tendency to slip through the cracks.
We want to build and ship the SDE to you all sooner, and more often alongside both Tranquility and Singularity releases. This would necessitate some changes to the SDE and its build process in order to achieve continuous delivery.
In the process of migrating static data from our legacy BSD content authoring tool to our new FSD authoring tool; we've been moving away from editing tables in a centralized SQL Server database, to YAML files tracked in revision control. This process is reflected by the SDE, having shipped both a SQL Server backup file and YAML files for some time now. We don't anticipate this migration to finish anytime soon.
One way of simplifying the build process can be achieved by naïvely converting the SQL tables to YAML files.
As an example, the last two entries of the eveUnits.yaml file would look like:
- unitID: 141 unitName: "Hardpoints" displayName: "hardpoints" description: "For various counts to do with turret, launcher and rig hardpoints" - unitID: 142 unitName: Sex displayName: "1=Male 2=Unisex 3=Female" description: null
The first entry of staStations.yaml would look like:
- stationID: 60000004 security: 0 dockingCostPerVolume: 0.0 maxShipVolumeDockable: 50000000.0 officeRentalCost: 10000 operationID: 26 stationTypeID: 1531 corporationID: 1000002 solarSystemID: 30002780 constellationID: 20000407 regionID: 10000033 stationName: "Muvolailen X - Moon 3 - CBD Corporation Storage" x: 1723680890880.0 y: 256414064640.0 z: -60755435520.0 reprocessingEfficiency: 0.5 reprocessingStationsTake: 0.05 reprocessingHangarFlag: 4
In order to smooth out such a transition, we would of course ship the good o' SQL Server backup file and the SQLite universe data alongside these new YAML files for the next one or two SDE releases for Tranquility. That way you'd have time to update any tools you may be running on top of the Static Data Export.
What are your thoughts and opinions on this proposal?
Thanks! |
|
|
CCP Tellus
C C P C C P Alliance
37
|
Posted - 2016.02.10 21:10:02 -
[2] - Quote
Zifrian wrote:If I read your post correctly, it sounds like you want to add just two more files to yaml? I only took those two tables as an example. The idea is to take every single table in the SQL Server backup file and convert them straight to YAML files.
One of the reasons behind this proposal is so that the entire SDE will end up consisting of just YAML files and nothing else. The other is to simplify the process of creating SDE releases so that we can release them more often with less manual labour.
There would be a one-to-one map between the columns of the SQL tables, and the columns of these YAML files. Similarly, there would be one YAML file for each SQL table that's currently provided in the database. That should hopefully minimize the effort needed for you all to modify existing conversion tools to conform to these proposed changes. |
|
|
CCP Tellus
C C P C C P Alliance
38
|
Posted - 2016.02.10 22:22:30 -
[3] - Quote
Zifrian wrote:Will the universe data also be migrated from SQLite? The universe data is authored as YAML files internally. These are then post-processed into an SQLite database. Why? I am not sure exactly. I'm contemplating just removing that post-processing step.
Zifrian wrote:Also, I get the impression that it would be easier to export the database prior to patch day? E.g. Citadel data before it is live? That's the idea! Automatically building SDE exports for Singularity as soon as changes are deployed there shouldn't be much trouble either with these changes. |
|
|
CCP FoxFour
C C P C C P Alliance
4291
|
Posted - 2016.04.25 21:48:44 -
[4] - Quote
Just for clarity, we plan on doing both the old and new style SDE for a few releases since we know it will take time for people to convert. This is why we didn't worry to much about putting out a pre-release version since we fully intended on doing both anyways for a bit.
@CCP_FoxFour // Technical Designer // Team Tech Co
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
|
CCP Tellus
C C P C C P Alliance
57
|
Posted - 2016.04.26 13:07:47 -
[5] - Quote
There's a new SDE release available on our developes' site:
https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160426-TRANQUILITY-legacy.zip |
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.26 14:45:14 -
[6] - Quote
Desmont McCallock wrote:1. Will this be the naming format for the SDE filename from now on? Is there any particular reason you're relying on the filename format remaining consistent? It may or may not change in the future.
Desmont McCallock wrote:2. Where do we find the 'translationTables' table? Do we have to create it on our own? The 'translationTables' table was an internal look-up table used to construct the 'trnTranslations' and 'trnTranslationColumns' tables. It had not gotten deleted after the SDE export script ran. Do you know of a use-case for that table?
Desmont McCallock wrote:3. Would you mind fixing the duplicate entries in blueprints.yaml file? (Can provide exact info if needed) I can take a look at blueprints.yaml. Can you send me the details? |
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.26 15:55:08 -
[7] - Quote
Desmont McCallock wrote:In my tool I have an algorithm that tries to detect if the user has dropped in the specific folder the zipped file or (s)he has extracted the content of the zipped file. If a zipped file is detected the tool unzips the content automatically. The algorithm does a match search depending on the file name (we don't want to unzip any zip file but only the SDE one) . You can use the following regular expression:
sde-(.+).zip We will probably not change the file naming scheme, but we may change where we host static data exports in the future.
Salvoxia wrote:previous SDE releases always contained some kind of version or build number (for the last one it was 117575 for example). I always used it to reference the exact SDE release. Will there be something similar for the new release format or will you just use the release date? Use the date and the branch name. |
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.26 16:11:45 -
[8] - Quote
Hel O'Ween wrote:Is this upposed to be the official SDE release for the Citadel expansion. If so, ramTypeRequirements seems to have some PK issues: Am I missing something? I am not able to find a table called ramTypeRequirements.
|
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.26 20:44:42 -
[9] - Quote
Desmont McCallock wrote:3. Would you mind fixing the duplicate entries in blueprints.yaml file? (blueprintTypeID: 41590 has typeID: 38 for manufacturing specified two (2) times with different quantity) This his how it appears in-game: "Required Input Materials" for that blueprint lists 3x Nocxium and 5x Nocxium. |
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.26 22:16:58 -
[10] - Quote
Desmont McCallock wrote:And from your experience is this correct? Should it be such? That's not for me to judge. I suggest you file a bug report and one of the game design teams will take a look at it. :)
|
|
|
|
CCP Tellus
C C P C C P Alliance
59
|
Posted - 2016.04.27 14:26:42 -
[11] - Quote
Desmont McCallock wrote:Edit2: Importing the dgmAtrributeTypes table from the yaml file, I notice that a clean up has been made. 994 entries have been removed including some that we use in EVEMon data files generator. Is this intentional? That was not intentional. I'll make a new build of the SDE with several fixes, should have it ready in a few hours. Thanks for your patience. :)
|
|
|
CCP Tellus
C C P C C P Alliance
60
|
Posted - 2016.04.27 21:11:54 -
[12] - Quote
Hello everyone! I published a new SDE that should hopefully contain fixes for all the reported issues. Please let us know if there are any more issues.
https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY.zip https://cdn1.eveonline.com/data/sde/tranquility/sde-20160427-TRANQUILITY-legacy.zip |
|
|
CCP Tellus
C C P C C P Alliance
60
|
Posted - 2016.04.29 10:05:17 -
[13] - Quote
Desmont McCallock wrote:region.staticdata for each region is missing the radius. Are we supposed to calculate that from the given info? constallation and solarsystem staticdata files include that info however. You'll need to calculate the radius. It should be the Euclidean distance between the region's 'min' and 'max', divided by the square root of two.
|
|
|
CCP Tellus
C C P C C P Alliance
60
|
Posted - 2016.04.29 10:44:23 -
[14] - Quote
Desmont McCallock wrote:Shouldn't the formula be radius = GêÜ((xMax - x)^2 + (yMax - y)^2 + (zMax - z)^2) ? The distance between 'min' and 'max' is the diameter. You should divide that by two to get the radius, but someone messed it up by putting the division by two inside the square root for each field. The end result is that you divide the diameter with the square root of two instead.
This is based on my understanding of quickly looking through the source that used to build the universe SQLite file. If the numbers don't check out, I'll take another stab at it for you. :) |
|
|
CCP Tellus
C C P C C P Alliance
60
|
Posted - 2016.04.29 15:33:43 -
[15] - Quote
Desmont McCallock wrote:I'm afraid that there is something fundamentally wrong with the formula you use to calculate the radius. Sorry for all the confusion. Took a stab at this and this is what I get:
From the universe data SQLite file, we have that the radius is 5.57645447335415e+16.
sqlite> select regionID, regionName, radius from mapRegions where regionName = 'A821-A'; regionID regionName radius ---------- ---------- -------------------- 10000019 A821-A 5.57645447335415e+16
By letting x1, y1, and z1 be the region's 'max' coordinates, and x2, y2, and z2 be the region's 'min' coordinates, we get:
>>> math.sqrt((x1 - x2)**2 + (y1 - y2)**2 + (z1 - z2)**2) / 2 5.576454473354151e+16
This matches the radius of the region in the legacy SDE. It also turns out I was wrong about dividing by the square root of two. Terribly sorry. I should test code before reaching a conclusion. :(
Hope this helps. |
|
|
CCP Tellus
C C P C C P Alliance
60
|
Posted - 2016.04.29 16:22:01 -
[16] - Quote
Desmont McCallock wrote:Never the less, even if it's wrong I guess it won't be fixed as it's been set to stone for years in the SDE, right? Pretty much.
|
|
|
|
|