| Pages: [1] 2 3 4 :: one page |
| Author |
Thread Statistics | Show CCP posts - 1 post(s) |
|
CCP Zirnitra
   |
Posted - 2009.11.25 09:59:00 -
[1]
As the launch of Dominion is getting ever closer, we are all gearing up for the new interesting things it brings. As a part of this, we are of course doing a new Static Data Dump for you, and to make the life easier on all our 3rd party developers out there, we are once again providing the dump in advance of the actual deployment.
However, in the spirit of providing you with the data as quickly as possible, there might still be some last minute changes to the contents of the data dump, however the main parts of the data should be correct. In case there will be further changes we will of course provide you with an up to date version of the data dump, to ensure you always have the correct data.
We are always preparing a new Icon dump for you as well, although this process is slightly slower, and will be available in the next 24-48 hours. I will update this thread and the toolkit page, as soon as we have any updates available for you, so make sure you check in here in the following days.
Now to what you have all been waiting for, the data dump:
http://ccp.vo.llnwd.net/o2/data/Dominion_1.0.0_30936_db.7z
As the smart reader would have noticed, we have switched the compression method from the regular Zip to using 7zip. You can download the 7zip program free of charge, but most of the main stream archivers today will be able to extract it. We have made the compression switch as 7z provides far better compression, which means less time spent downloading for you and more time working with the data.
Enjoy!
|
|
Kravek
Lamb Federation Navy
C0VEN
|
Posted - 2009.11.25 10:03:00 -
[2]
\o/
Nice work.
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2009.11.25 10:16:00 -
[3]
yay
Win a Wyvern mothership for 10M ISK |
|
Entity
X-Factor Industries
Synthetic Existence
|
Posted - 2009.11.25 10:24:00 -
[4]
*purrs*
_
Got Item? | EVE API? | Cache? |
Vessper
SI Radio
|
Posted - 2009.11.25 11:30:00 -
[5]
Awesome!
EveHQ Character App |
Midge Mo'yb
Antares Shipyards
Phalanx Alliance
|
Posted - 2009.11.25 13:10:00 -
[6]
DUMP? EARLY?
OBVIOUS TARP IS OBVIOUS!!
DO NOT DOWNLOAD!!
-----------------------------------------------
|
John Zorg
Caldari
The Arrow Project
Morsus Mihi
|
Posted - 2009.11.25 15:30:00 -
[7]
Time to do some more dev work it seems. Thanks for the new content.
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.11.25 17:40:00 -
[8]
Edited by: Amida Ta on 25/11/2009 17:46:05
What happened to typeActivityMaterials? Is the data still available somewhere?
I've seen invTypeMaterials, but that doesn't seem to contain everything from before.
Edit: Seems some stuff is in ramTypeRequirements.
_________________________
EveAI.Live - The EVE-Online API/class library for .Net, C# and VB.Net |
Vanessa Vansen
|
Posted - 2009.11.25 21:20:00 -
[9]
yay ... now I only have to convert it into something I can use ... or wait till somebody else has done so
|
Cyril
Faeneror Lupus
One Stop Research
|
Posted - 2009.11.25 21:43:00 -
[10]
This is very awesome to have the dump before it goes live. I had been thinking about working on some web based tools and this is a great motivational boost =) (now if only I can find some spare time)
I just updated the eve-dev wiki page to link to this post and the dump file hopefully someone has the time to start kicking out the conversions soon.
-----------------------------------------
insert witty sig here. |
Epitrope
The Citadel Manufacturing and Trade Corporation
|
Posted - 2009.11.26 00:08:00 -
[11]
 Originally by: Amida Ta
Edited by: Amida Ta on 25/11/2009 18:36:01
What happened to typeActivityMaterials? Is the data still available somewhere?
I've seen invTypeMaterials, but that doesn't seem to contain everything from before.
Edit: Seems stuff is in ramTypeRequirements.
This release of the data export has build requirements split into two tables, invTypeMaterials and ramTypeRequirements. The old table, typeActivityMaterials, was a view to these tables, and not a very accurate or useful one.
invTypesMaterials contains all the reprocessing data, including the oft-requested ore reprocessing values. Everything that an item type reprocesses to is also used to manufacture that type, and these materials are affected by waste. If you're coming from a blueprint-oriented mindset, note that this table is based on the productTypeID, not the blueprintTypeID. Also, don't forget to take into account invTypes.portionSize.
ramTypeRequirements includes all the blueprint-oriented stuff: extra build materials (which are unaffected by waste), research materials such as R.Db., skill requirements for building or researching, datacores for inventing or reverse engineering, and so on. Note that this table is based on the blueprintTypeID.
Depending on how you've used the previous data exports, it might be easier for you to denormalize the data into a single table, based on blueprintTypeID, which shows all the materials needed by a blueprint and how much of each is affected by waste. I've put that query up on the eve-dev wiki.
To those responsible, thank you for dropping typeActivityMaterials in favor of itm/rtr!
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.11.26 00:17:00 -
[12]
Thanks for all your time, but did already figure that out (maybe it can still help somebody else with a similar question).
In fact I like the new organization better, too.
Did even change the relevant EveAI datastructures into the new form.
_________________________
EveAI.Live - The EVE-Online API/class library for .Net, C# and VB.Net |
Sidrat Flush
Caldari
Life is Experience
New Eden Hardware Emporium
|
Posted - 2009.11.26 04:21:00 -
[13]
Originally by: Epitrope
This release of the data export has build requirements split into two tables, invTypeMaterials and ramTypeRequirements. The old table, typeActivityMaterials, was a view to these tables, and not a very accurate or useful one.
invTypesMaterials contains all the reprocessing data, including the oft-requested ore reprocessing values. Everything that an item type reprocesses to is also used to manufacture that type, and these materials are affected by waste. If you're coming from a blueprint-oriented mindset, note that this table is based on the productTypeID, not the blueprintTypeID. Also, don't forget to take into account invTypes.portionSize.
ramTypeRequirements includes all the blueprint-oriented stuff: extra build materials (which are unaffected by waste), research materials such as R.Db., skill requirements for building or researching, datacores for inventing or reverse engineering, and so on. Note that this table is based on the blueprintTypeID.
Depending on how you've used the previous data exports, it might be easier for you to denormalize the data into a single table, based on blueprintTypeID, which shows all the materials needed by a blueprint and how much of each is affected by waste. I've put that query up on the eve-dev wiki.
To those responsible, thank you for dropping typeActivityMaterials in favor of itm/rtr!
This information is useful and I thank you for passing it on, it will now be so much easier to calculate waste properly and it can be called Raw and Extra instead of (hopefully as I haven't seen the actual data yet) activity 1 and activity 6 which is what eve industrial organiser was doing.
I do hope they haven't messed about with the manufacturing formula though as that would be annoying.
Eve-online Industrial Organiser thread full batch manufacturing
|
Jercy Fravowitz
School of Applied Knowledge
|
Posted - 2009.11.26 09:04:00 -
[14]
usual barrage of conversions:
mysql singlefile (you are most likely looking for this one)
this one should also load just fine in php-webloader-scripts, if it whines about something with timestamps at the end, ignore it.
sqlite singlefile
postgres singlefile
sqlite "EVE Asset Manager" edition
less popular:
MSsql script dump (useful for those who cba to upgrade to sql server 2008)
mysql "one file per table" dumps
mysql "one file per table" dumps as tarball
there may or may not be other flavors i forgot about if you look here.
as usual,
- if you need help with any of this,
- want custom dumps (missing sql flavors, xml, csv),
- want the less than official tables,
- or want to complain about issues,
please show up on #eve-dev on irc.coldfront.net and whine loud enough for me to hear it.
|
Cyril
Faeneror Lupus
One Stop Research
|
Posted - 2009.11.26 09:53:00 -
[15]
Originally by: Jercy Fravowitz
usual barrage of conversions:
mysql singlefile (you are most likely looking for this one)
this one should also load just fine in php-webloader-scripts, if it whines about something with timestamps at the end, ignore it.
sqlite singlefile
postgres singlefile
sqlite "EVE Asset Manager" edition
less popular:
MSsql script dump (useful for those who cba to upgrade to sql server 2008)
mysql "one file per table" dumps
mysql "one file per table" dumps as tarball
there may or may not be other flavors i forgot about if you look here.
as usual,
- if you need help with any of this,
- want custom dumps (missing sql flavors, xml, csv),
- want the less than official tables,
- or want to complain about issues,
please show up on #eve-dev on irc.coldfront.net and whine loud enough for me to hear it.
You are my hero. I just spent the last 6 hours trying to get mssql 2008 express to export to myslq with the odbc and was about to start pulling out my hair.
-----------------------------------------
insert witty sig here. |
Sidrat Flush
Caldari
Life is Experience
New Eden Hardware Emporium
|
Posted - 2009.11.26 10:29:00 -
[16]
Originally by: Cyril
You are my hero. I just spent the last 6 hours trying to get mssql 2008 express to export to myslq with the odbc and was about to start pulling out my hair.
Ahh damn it, I've already got the ODBC connection up and running and I've put off going to sleep as I'm still playing with it.
It's damn tricky to get the manufacturing data (raw and extra material) to link up properly though, still I'll get there, then I can start creating new links to the spreadsheet.
Eve-online Industrial Organiser thread full batch manufacturing
|
Jercy Fravowitz
School of Applied Knowledge
|
Posted - 2009.11.26 12:04:00 -
[17]
Originally by: Sidrat Flush
Ahh damn it, I've already got the ODBC connection up and running and I've put off going to sleep as I'm still playing with it.
It's damn tricky to get the manufacturing data (raw and extra material) to link up properly though, still I'll get there, then I can start creating new links to the spreadsheet.
a bit of timeline ...
09:59 CCP Zirnitra posts link on forum (this thread)
10:20 first 404 in my http access log of someone requesting /dom100/ dir.
16:58 someone pastes me the link
17:00 remote powerup and vnc into eve machine, restore into mssql, export as 1.2GB utf16 mssql script
17:30 data moved to linux box, conversion to utf8 mssql script (600MB)
this is followed by about 2 hours of turdpolishing: fixing up column types
for the new tables, making sure stuff is where it is supposed to be.
another hour for crawling home, all up to this was done remotely while at the office.
21:00 at home, some more fiddling, getting annoyed by the old machine being slow.
22:00 installing postgres onto the new db server, getting annoyed by postgres user management.
at this point the mysql and sqlite conversions are already available.
23:00 some benchmarking of the conversion process.
on the new machine, the whole mssql->mysql->(pgsql,sqlite) process including bzipping the output takes about 42 minutes, 32 of which are consumed by just postgres being slow.
so the most relevant work really takes all of about 10 minutes while watching a script run. what took longer in this case was the turdpolishing, because there were new turds, aeh, i mean tables.
again, if anyone is having issues with the new materials data or anything else in the dump and conversions, please show up in #eve-dev on irc.coldfront.net
we do have conversions available that result in data equivalent to the old typeActivityMaterials view, but would suggest to only use that for unmaintained legacy applications.
there are cases that are not handled by the view, and the new data makes more sense in general.
|
Wollari
Phoenix Industries
Wicked Nation
|
Posted - 2009.11.26 13:10:00 -
[18]
woah. nice
|
Sidrat Flush
Caldari
Life is Experience
New Eden Hardware Emporium
|
Posted - 2009.11.26 14:22:00 -
[19]
Could someone PLEASE explain the steps required to get the full list of input items to manufacture from a blueprint now as my head is spinning.
Going out for some fresh air, clear my head and mull it over, hope there'll be a helpful post waiting for when I get back in an hour or so, otherwise it's back to scratching my head again.
|
Catari Taga
Centre Of Attention
Rough Necks
|
Posted - 2009.11.26 15:42:00 -
[20]
Originally by: Jercy Fravowitz
a bit of timeline ...
09:59 CCP Zirnitra posts link on forum (this thread)
10:20 first 404 in my http access log of someone requesting /dom100/ dir.
Well if CCP can release ahead of time, why can't you? j/k, thanks for the conversion!
|
Vanessa Vansen
|
Posted - 2009.11.26 15:53:00 -
[21]
Originally by: Jercy Fravowitz
usual barrage of conversions:
mysql singlefile (you are most likely looking for this one)
this one should also load just fine in php-webloader-scripts, if it whines about something with timestamps at the end, ignore it.
sqlite singlefile
My hero, too!
|
Jercy Fravowitz
School of Applied Knowledge
|
Posted - 2009.11.26 16:58:00 -
[22]
Originally by: Sidrat Flush
Could someone PLEASE explain the steps required to get the full list of input items to manufacture from a blueprint now as my head is spinning.
example:
| typeID | typeName | portionSize |
| 27922 | Lockbreaker Bomb | 20 |
| 27923 | Lockbreaker Bomb Blueprint | 1 |
you can find the relation between blueprint and product also in invBlueprintTypes, pay attention to blueprintTypeID and productTypeID there.
now, for the materials ... first you look up the PRODUCT in invTypeMaterials:
mysql> select m.*,t.typeName from invTypeMaterials m, invTypes t where t.typeID=m.materialTypeID and m.typeID=27922;
| typeID | materialTypeID | quantity | typeName |
| 27922 | 34 | 10000 | Tritanium |
| 27922 | 35 | 100 | Pyerite |
| 27922 | 36 | 10 | Mexallon |
| 27922 | 37 | 10 | Isogen |
| 27922 | 38 | 10 | Nocxium |
| 27922 | 39 | 10 | Zydrine |
| 27922 | 40 | 10 | Megacyte |
this is what you will get out when you refine/reprocess portionSize of the product. this is what ME waste is added to, and reprocessing skill loss removed from. note you can also find ore, ice and named stuff in that table, we finally get all that in one place. this is good.
now, for manufacturing, to that base-plus-MEwaste you add what you get when you look up the BLUEPRINT and the right activity (1: Manufacturing) in ramTypeRequirements:
mysql> select r.*,t.typeName from ramTypeRequirements r, invTypes t where t.typeID=r.requiredTypeID and r.typeID=27923;
| typeID | activityID | requiredTypeID | quantity | damagePerJob | recycle | typeName |
| 27923 | 1 | 34 | 1096272 | 1 | 0 | Tritanium |
| 27923 | 1 | 35 | 500164 | 1 | 0 | Pyerite |
| 27923 | 1 | 36 | 47939 | 1 | 0 | Mexallon |
| 27923 | 1 | 37 | 9106 | 1 | 0 | Isogen |
| 27923 | 1 | 38 | 1812 | 1 | 0 | Nocxium |
| 27923 | 1 | 39 | 1226 | 1 | 0 | Zydrine |
| 27923 | 1 | 40 | 277 | 1 | 0 | Megacyte |
| 27923 | 1 | 3380 | 1 | 0 | 0 | Industry |
and yes, this looks like suck and fail when posted here. the power of proportional fonts. yay.
HTH.
|
Wyehr
Rage of Inferno
|
Posted - 2009.11.26 18:49:00 -
[23]
Jercy, thanks for the examples. Just one quick note:
The ME waste seems to be applied to both the invTypeMaterials quantity and the ramTypeRequirements quantity, not just the one. Using your example, a ME0 Lockbreaker blueprint requires 1216899 Trit, which is exactly 1.1 * (10,000 + 1,096,272).
That is, unless the new expansion is going to change the way manufacturing works, or change the lockbreaker BPO.
|
Jercy Fravowitz
School of Applied Knowledge
|
Posted - 2009.11.26 20:11:00 -
[24]
Originally by: Wyehr
The ME waste seems to be applied to both the invTypeMaterials quantity and the ramTypeRequirements quantity, not just the one. Using your example, a ME0 Lockbreaker blueprint requires 1216899 Trit, which is exactly 1.1 * (10,000 + 1,096,272).
check actual manufacturing requirements.
the values shown by showinfo on the blueprint are plain incorrect.
https://bugs.eve-online.com/files/3079911954.png
defect on that is resolution pending.
i used bombs as an example because that is one case where you did not stand a chance to figure out actual manufacturing requirements under the old typeActivityMaterials data model.
basicly that simplified model is what showinfo on the blueprint uses, but actual manufacturing uses the split-tables model.
|
Sprobe
Factorization 42 Corp
|
Posted - 2009.11.26 22:43:00 -
[25]
Originally by: Jercy Fravowitz
Originally by: Wyehr
The ME waste seems to be applied to both the invTypeMaterials quantity and the ramTypeRequirements quantity, not just the one. Using your example, a ME0 Lockbreaker blueprint requires 1216899 Trit, which is exactly 1.1 * (10,000 + 1,096,272).
check actual manufacturing requirements.
the values shown by showinfo on the blueprint are plain incorrect.
https://bugs.eve-online.com/files/3079911954.png
defect on that is resolution pending.
i used bombs as an example because that is one case where you did not stand a chance to figure out actual manufacturing requirements under the old typeActivityMaterials data model.
basicly that simplified model is what showinfo on the blueprint uses, but actual manufacturing uses the split-tables model.
yea, Jercy is plain right - we did a lik late night chat on that a few weeks back on #eve-dev and actually.. this is the first database dump which gives you the possibility to get the right ME waste according to manufacturing quotes - forget whatever the BPOs / BPCs say.. check the manufacturing quotes.
never to old to play |
Wyehr
Rage of Inferno
|
Posted - 2009.11.26 22:52:00 -
[26]
Wow, that's crazy. Hope they managed to get it fixed in the next release.
|
Sidrat Flush
Caldari
Life is Experience
New Eden Hardware Emporium
|
Posted - 2009.11.27 00:09:00 -
[27]
Unfortunately it's working as intended and always has been.
The fact they've split up the tables means to nubbins here at least is that they've over normalized to hell in a pizza box and I'm STILL trying to figure out how to get both sets of information to wit - Base minerals and Raw minerals - to appear on the column row in a cross table query.
I'm certain it can't be done in one query so if anyone would like to share the multiple steps it would be appreciated.
Eve-online Industrial Organiser thread full batch manufacturing
|
Celebrain
1st Steps Academy
Tread Alliance
|
Posted - 2009.11.27 14:52:00 -
[28]
It's not "over normalized" :P Thank you CCP for improving it, for giving us more data in an improved table layout, and thank you for giving it to us early to give us time to get our stuff in line with it. Keep up the good work.
|
Salvis Tallan
Gallente
Bunnie's Boys
|
Posted - 2009.11.27 16:36:00 -
[29]
WoooHooo! Exactly what I wanted! Thanks for the good work ccp, and lets make a habit out of this.
------
|
Matthew
Caldari
BloodStar Technologies
|
Posted - 2009.11.27 20:51:00 -
[30]
Edited by: Matthew on 27/11/2009 20:54:50
While Jercy's method looks fine to me for T1 manufacturing, it seems to be slightly more complicated for T2.
For example, looking at the Tritanium requirement to build 1 unit of Medium Shield Transporter II.
invTypeMaterials will give you a figure of 1660 tritanium. However, the actual job quote asks for just 336 Tritanium.
The difference is due to the requirement for the T1 module. Looking up the T1 version in invTypeMaterials indicates that the T1 module requires 1355 tritanium.
1660-1355 = 305. Adding the 10% ME wastage on top gets you to the 336 Tritanium requested by the quote.
The key to this is the recycle field in ramTypeRequirements. This field is 1 for the T1 module requirement, but 0 for everything else. Which gives us the differentiation as to why we have to remove the Tritanium in the T1 module, but not that in the R.A.M. - Shield Tech.
So it looks like we actually need three elements to make up the full manufacturing requirement:
(1) Records from invTypeMaterials for typeID=productTypeID
(2) Records from ramTypeRequirements for activityID=1, typeID=blueprintTypeID
(3) Records from invTypeMaterials for typeID=requiredTypeID from (2) where recycle=1
( (1)-(3) )*wasteFactor then becomes your Raw Materials
(2) becomes your Extra Materials and Skills (differentiated by the categoryID of the requiredTypeID)
Originally by: Sidrat Flush
I'm certain it can't be done in one query so if anyone would like to share the multiple steps it would be appreciated.
My approach may well not be the most elegant, and I haven't actually done it to check it works yet, and it works on the assumption that for the final output you don't care whether the requirement is Raw Material, Extra Material or skill. My aim with this is to get something as close to what I got out of the old tables as possible. Here's what I have in mind:
A) 3 queries producing the three data elements identified above, with the exception that the query for (3) returns the negative of the quantity value
B) Union datasets (1) and (3)
C) Group B by materialTypeID, sum by quantity - this is your pre-waste Raw Material quantity
D) Apply waste to C
E) Union D with (2) - treating materialTypeID and requiredTypeID as the same field
F) Group E by requiredTypeID, sum by quantity - this is your final output
Edit: and clearly at some point you'll have to choose to go with blueprintTypeID or productTypeID, and just convert the other over using invBlueprintTypes, otherwise the union in step E won't work properly.
-------
There is no magic Wand of Fixing, and it is not powered by forum whines. |
| |
|
| Pages: [1] 2 3 4 :: one page |
| First page | Previous page | Next page | Last page |