Pages: [1] 2 3 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 10 post(s) |
|
CCP Guard
C C P C C P Alliance
2211
|
Posted - 2012.05.03 16:41:00 -
[1] - Quote
Our EVE 3rd party development community is simply awe inspiring and now that I've gotten that out of my system, I recommend all 3rd party developers go here and read this dev blog about some changes we're making to the EVE 3rd Party Toolkit. CCP Guard | EVE Community Developer |-á@ccp_guard |
|
Cathrine Kenchov
Ice Cold Ellites
2
|
Posted - 2012.05.03 16:46:00 -
[2] - Quote
cool stuff |
Steve Ronuken
Fuzzwork Enterprises
392
|
Posted - 2012.05.03 16:47:00 -
[3] - Quote
/me cries quietly about more conversion work
Yay! Time to learn something new. (that's actually a positive yay, rather than a sarcastic one)
Now to wait for the update so I can download it. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities. |
Akrasjel Lanate
Black Thorne Corporation Black Thorne Alliance
719
|
Posted - 2012.05.03 16:49:00 -
[4] - Quote
Cool but short |
Packtu'sa
Nabaal Construction and Industrials Corp Nabaal Syndicate
0
|
Posted - 2012.05.03 16:54:00 -
[5] - Quote
Can you elaborate on the decision to change to YAML? My understanding of the format (having used it before) is that it's appropriate when the information needs to be both human- and machine-readable. Configuration files are an obvious candidate. Static data, on the other hand, seems more appropriately stored in a database or at least a table format (like CSV). YAML is a lot of key/value pairs and arrays. Why was it chosen to represent tabular data? |
Droxlyn
TOHA Heavy Industries TOHA Conglomerate
78
|
Posted - 2012.05.03 16:59:00 -
[6] - Quote
I'm confused as well. Why not keep the data in the database and export it at each release for the client to use?
For those that do not want to change, it'll just require ramming it back into the database somehow. |
ctx2007
Wychwood and Wells
61
|
Posted - 2012.05.03 17:05:00 -
[7] - Quote
sixth |
darmwand
Repo.
37
|
Posted - 2012.05.03 17:14:00 -
[8] - Quote
Aaah, this is awesome! Thanks a lot, this should make it much easier to deal with EVE data, using YAML is definitely the right thing to do. darmwand Repossession Agent http://www.repo-corp.net/ Recruitment is OPEN |
Droxlyn
TOHA Heavy Industries TOHA Conglomerate
78
|
Posted - 2012.05.03 17:40:00 -
[9] - Quote
Samples for those too lazy to open it up:
graphicsIDs.yaml wrote: 12: description: Stargate graphicFile: res:/Model/Jumpgate/Caldari/cj2/cj2.blue obsolete: true
typeIDs.yaml wrote: 5: graphicID: 6 6: graphicID: 1015
I hope they find more value in later YAMLs. These could have been done just as well with CSV files. |
Antihrist Pripravnik
Scorpion Road Industry
6
|
Posted - 2012.05.03 17:59:00 -
[10] - Quote
Never heard of YAML, but I've heard (and worked) a lot with JSON. Why YAML and not JSON? CCP Ytterbium: Yarrblblbgrlblbgrlblblblbblbgrlblblbgrblblyarrrrdrooooooolonthekeyboardlikealunatic |
|
Thebriwan
LUX Uls Xystus
41
|
Posted - 2012.05.03 18:08:00 -
[11] - Quote
I truly don't understand why anyone would use this Yam-something over XML but that is not my decision to make.
Where I really see a problem is here:
There are at least two totally different uses cases for the static data:
a) Something like a stand alone application for processing manually some data.
b) A web app processing Millions of datasets and calculation on the fly.
For a) it does not really matter if there are csv oder xml or yam-something files.
But b) needs a database.
And for all the effort you are doing here there is no way to provide sql files anymore? |
Two step
Aperture Harmonics K162
1926
|
Posted - 2012.05.03 18:10:00 -
[12] - Quote
A suggestion:
Before you guys switch over to the new system, how about you release a *full* data dump in the new format? Giving us 2 sample files, especially when it doesn't sound like there will be a 1-1 mapping of old tables to the new files isn't enough for people to build tools to handle the new data.
Edit: I just re-read the blog, sounds like you are *only* changing those two files for Inferno. My request stands for when you change over more stuff.. :) CSM 7 Secretary CSM 6 Alternate Delegate @two_step_eve on Twitter My Blog
|
Packtu'sa
Nabaal Construction and Industrials Corp Nabaal Syndicate
0
|
Posted - 2012.05.03 18:18:00 -
[13] - Quote
Antihrist Pripravnik wrote:Never heard of YAML, but I've heard (and worked) a lot with JSON. Why YAML and not JSON?
YAML is a superset of JSON. That said, I don't think either are appropriate. They can handle tabular data, but that's like saying you can send someone a picture as a Word document filled with comma-separated color values. Yes, the information is there, but it's in the wrong format.
(As far as I know,) EVE static data is tabular, not flexible key/value data like YAML is meant to represent. XML is sort of in between the two.
Honestly, I'm more concerned that CCP is choosing to use YAML internally than that they're forcing us to use it. I guess I shouldn't jump to conclusions before CCP can comment on it, though. |
Xarrg
Crushed Ambitions Reckless Ambition
3
|
Posted - 2012.05.03 18:43:00 -
[14] - Quote
How easy/hard to transport them back to sql format ? I'm sure most of the 3rd party guys will stick with they ms/mysql, so this will be just a extra step for us.
|
Zagdul
Clan Shadow Wolf Fatal Ascension
580
|
Posted - 2012.05.03 18:50:00 -
[15] - Quote
Item ID please stop using names.
It's not rocket surgery. |
Vessper
Eve Engineering Finance Eve Engineering
8
|
Posted - 2012.05.03 18:51:00 -
[16] - Quote
How long before we stop getting the MSSQL format?
|
Abdiel Kavash
Paladin Order Fidelas Constans
435
|
Posted - 2012.05.03 18:55:00 -
[17] - Quote
Because relational databases which have decades of research, optimalizations, and development behind them are too mainstream. Let's throw XML at the problem.
Also, we want to make life easy for application programmers, so for the forseeable future we will make you pull half of the data from a DB, and half of it from XML. You know, just to expand your horizons. |
|
Chribba
Otherworld Enterprises Otherworld Empire
3365
|
Posted - 2012.05.03 18:56:00 -
[18] - Quote
Vessper wrote:How long before we stop getting the MSSQL format?
This?
I'd probably be looking at converting it all back to db since that's what I prefer myself though.
/c
|
|
|
CCP Solomon
C C P C C P Alliance
142
|
Posted - 2012.05.03 19:01:00 -
[19] - Quote
As some have correctly noted, the reason for the split format delivery is due to an internal process change in how we manage static data, both during authoring and at run-time. This is a gradual migration effort that will see more and more portions of the static data dump delivered as YAML files. There is currently no date for when we will stop delivering the database dump, rather it will occur when there is no data left in it.
We will be delivering up to date samples of the YAML files and data base dumps ahead of each major release (assuming the data has changed) to give 3rd party developers a chance to update their tools.
These will be posted on the EVE Toolkit page. Associate Technical Producer - Foundation Technology |
|
|
CCP Solomon
C C P C C P Alliance
142
|
Posted - 2012.05.03 19:03:00 -
[20] - Quote
Chribba wrote:Vessper wrote:How long before we stop getting the MSSQL format?
This? I'd probably be looking at converting it all back to db since that's what I prefer myself though. /c
Yes, we absolutely encourage this and anticipated it to a certain extent, the 3rd party developer community are a resourceful bunch.
Associate Technical Producer - Foundation Technology |
|
|
James Bryant
Deep Core Mining Inc. Caldari State
7
|
Posted - 2012.05.03 19:18:00 -
[21] - Quote
CCP Solomon wrote:As some have correctly noted, the reason for the split format delivery is due to an internal process change in how we manage static data, both during authoring and at run-time. This is a gradual migration effort that will see more and more portions of the static data dump delivered as YAML files.
Can you shed any light on what this process change is and why it was undertaken? That might help us select the correct tools to manage a hybrid YAML/SQL environment like what I'm assuming you guys will be doing.
|
Andrea Griffin
274
|
Posted - 2012.05.03 19:26:00 -
[22] - Quote
I'm a bit befuddled over the choice of YAML but hey, if it works for you guys then that's great. I know that a lot of guys where I work are moving a lot of our configs over from a simple key=value format to YAML.
I'm not sure I would want it for the massive data sets that Eve uses, but it's a standard format, it isn't XML (which is good and bad), and it's very readable. So, what'ev. : > I'm just happy that CCP is awesome enough to provide us with the data in the first place. CCP Sreegs is my favorite developer. |
Ciar Meara
Virtus Vindice
643
|
Posted - 2012.05.03 19:49:00 -
[23] - Quote
casual corpse collector (wait, what?)
I love my corpses, but now I can love em even more ... - [img]http://go-dl1.eve-files.com/media/corp/janus/ceosig.jpg[/img] [yellow]English only please. Zymurgist[/yellow] |
Packtu'sa
Nabaal Construction and Industrials Corp Nabaal Syndicate
1
|
Posted - 2012.05.03 19:58:00 -
[24] - Quote
CCP Solomon wrote:As some have correctly noted, the reason for the split format delivery is due to an internal process change in how we manage static data, both during authoring and at run-time.
Yes, but why? In what way is the data changing such that YAML is the most appropriate format, and/or in what way is YAML the most appropriate format for the existing data?
There may well be a very good reason. I'm just interested in hearing it. |
Matthew
BloodStar Technologies
2
|
Posted - 2012.05.03 20:05:00 -
[25] - Quote
CCP Solomon wrote:As some have correctly noted, the reason for the split format delivery is due to an internal process change in how we manage static data, both during authoring and at run-time. This is a gradual migration effort that will see more and more portions of the static data dump delivered as YAML files.
The blog itself seems to suggest that only the client is currently using the static data in YAML format. Does this mean that the server (fed as it is through a lovely beast of an SQL Server), still has this data in a database format?
If so, what is the logic behind not providing all the data in both formats?
Or is the plan that all static data will exist only as YAML throughout both client and server?
My concern with this move is that while, as you point out, there are plenty of yaml readers for common programming languages, support for it in more off-the-shelf usage scenarios (ranging from an SQL Express install on someone's desktop, right down to someone knocking together their own home-brew spreadsheet) is rather less complete. Unfortunately, the latter group of data-dump users are unlikely to really even consider themselves 3rd party developers, so the first we'll probably hear from them is the wave of moans when the first of the really key data tables transitions over (the rest of invTypes, for example).
I guess I'd just be happier at accepting the higher barrier to entry that this creates if there was a bit more detail as to the advantages you expect in moving the data to YAML-only. Right now it looks like a shift of format of what is essentially perfectly happy, tablular, relational data, without any obvious benefit.
Chribba wrote: I'd probably be looking at converting it all back to db since that's what I prefer myself though.
If CCP are really going to push the data as YAML-only, then I can see this being a very popular service, particularly with myself! |
|
CCP Redundancy
C C P C C P Alliance
26
|
Posted - 2012.05.03 20:20:00 -
[26] - Quote
I figure I'll just answer some questions in an incomprehensible techy way.
As an organization, CCP has decided that we benefit from developers being able to work in branches (in a Perforce sense), and working in branches eventually means that you need to be able to change your data with your code and not affect other people. Binary formats and DBs can be perfect for the run-time data requirements that you have, but they're sucky from the point of view of understanding and merging data as a frail human doing an integration. So we want our data in files, and we want to be able to merge them.
So why not CSV? Yes, it would seem to be more appropriate to some of the data we've shown off so far, but at some point you also have to realize that the reason your data is tabular is because you've been storing it in a tabular storage medium. It's also really sucky to deal with foreign key relations between text files. As we progressively migrate more data, and deal with things like moons being the children of planets, we can decide to represent that in the structure of our data.
And why not JSON? Well, JSON is spiffy and all, but we already use YAML as an institution (if you've ever looked at our .red files for our ship assets etc), and it can deal with some things much more nicely than JSON (we use YAML reference support for some things). JSON somewhat suffers for having been built for a language that hasn't had a standard concept of a map until ECMA script 6 (stringified attribute names on objects just don't count, and this issue carries over into the proposed JSON schema validation standard).
We can output our static data as JSON, it's just not what we want to work with. We asked about this at the fanfest, but I think the detail was probably missed in the noise of the orbital bombardment
So we wanted to use YAML, and a no-sql-ish document-like data setup, but this isn't really appropriate for the runtime. In fact, we don't use it for the runtime... we use a structured binary format that's built from the data (think MessagePack), attempts to minimize memory overhead and disk seek operations. In some cases, we might even use Sqlite (woo, standard python library!) as appropriate for the use-case.
The end result of what we're trying to achieve is better runtime memory overhead performance, with an easier time for our developers to add / remove and change data formats in a human-understandable base format that can be versioned with changes to the code that are associated with it and isolated in branches. Some of the data will get built straight back into relation tables in MS-SQL, but some of it will just sit in a structured format that means that all related data to a particular "thing" is just accessible right there without subsequent join operations or lookups being required. We get to be able to sync our source control and get our data as it was at that point.
An important thing for us is to just try this and see if it can work for us, which is why we're starting so small. Beyond the changes to the data are much bigger changes to our tools and methods of working, not so much based on what the format is, but more on the issue of the data being local, isolated and in files rather than a central authoring DB.
I would suggest that the best long-term solution *might* be to look at NoSQL type databases, of which there are a number of free and very good options, or you can choose to try and maintain scripts that process our data into relational structures. |
|
Dalmont Delantee
The Black Legionnares SpaceMonkey's Alliance
27
|
Posted - 2012.05.03 20:32:00 -
[27] - Quote
Wow, that is seriously nerdgasm speak...I understood about 1 out of 10000 words but still made me shiver :P
|
James Bryant
Deep Core Mining Inc. Caldari State
8
|
Posted - 2012.05.03 20:39:00 -
[28] - Quote
Thanks CCP Redundacy,
We too are dealing with the difficulties of maintaining some kind of versioning system with DDL and static data; it is really a problem that doesn't have a good solution yet.
I can understand going to files that are easily parseable by CVS/Git/Whatever. It also allows people to check out the files and make their own changes locally without affecting the test database, and without having to load a fresh database copy into their local test environment every time they need something.
VS2010 Premium actually has some good SQL versioning capabilities when used with Team Foundation Server, but I have to say that I'm intrigued by CCP's approach here. Kinda the best of both worlds, in a certain sense (for you guys, a bit less so for us). A bit unwieldy, for multiple join type queries, but that stuff can be handled in code instead of in the database, I suppose. |
|
CCP Nobody
Royal Amarr Institute Amarr Empire
0
|
Posted - 2012.05.03 21:02:00 -
[29] - Quote
Matthew wrote: The blog itself seems to suggest that only the client is currently using the static data in YAML format. Does this mean that the server (fed as it is through a lovely beast of an SQL Server), still has this data in a database format?
No, we will also change the servers static data to YAML because, as CCP Redundancy said, we are moving away from the central authoring DB solution.
The gain in this is that the client will not contain data that it doesn't need and neither will the server, which can't be bad |
|
Packtu'sa
Nabaal Construction and Industrials Corp Nabaal Syndicate
1
|
Posted - 2012.05.03 21:28:00 -
[30] - Quote
Cheers. To clarify, does CCP have any plans to add structures which can't easily be represented in a database? (I'm having difficulty imagining what these might be, but YAML can do a lot.)
If there are any performance issues with YAML in third-party applications, I'm sure someone over at the Technology Lab will come up with a more useful package. (Something similar to the binary format that CCP Redundancy mentioned?)
[EDIT]
CCP Redundancy wrote:I figure I'll just answer some questions in an incomprehensible techy way. This, please, more of this! I've recently come back to EVE after playing some other in-development games, and it's refreshing to once again chat with devs who respect the player base and are themselves respectable. |
|
|
|
|
Pages: [1] 2 3 :: one page |
First page | Previous page | Next page | Last page |