EVE Search - Tyrannis 1.0.4 Static DataDump and ImageDump

All Channels

EVE Technology Lab

Tyrannis 1.0.4 Static DataDump and ImageDump

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: [1] 2 :: one page
Author	Thread Statistics \| Show CCP posts - 1 post(s)

CCP Zirnitra	Posted - 2010.08.31 11:46:00 - [1] With a slight delay, here are the Tyrannis 1.0.4 data and image dumps. Static Data Dump: http://content.eveonline.com/data/Tyrannis_1.0.4_33864_db.7z Image Dump: http://content.eveonline.com/data/Tyrannis_1.0.4-imgs.7z Be aware that this new data dump is not 100% backwards compatible on a few tables. -- NON BACKWARDS COMPATIBLE CHANGES BELOW! -- Adding table eveIcons for 2D graphics. -- Table eveGraphics recreated for 3D graphics alone. -- Removing graphicID (3D) and adding iconID (2D) for the following tables: -- mapLandmarks, chrAncestries, chrAttributes, chrBloodlines, chrRaces, dgmAttributeTypes, -- dgmEffects, invCategories, invGroups, invMarketGroups, invMetaGroups -- tables with iconID (2D) added: -- invTypes, chrFactions, crpNPCCorporations -- tables that retained graphicID: -- invTypes

Lutz Major Austriae Est Imperare Orbi Universo	Posted - 2010.08.31 12:40:00 - [2] Wooot! :) PS: Will there be any changes for the upcomming 1.0.5 version?
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.08.31 16:35:00 - [3] conversions: mysql singlefile (you are most likely looking for this one) this one should also load just fine in php-webloader-scripts, if it whines about something with timestamps at the end, ignore it. sqlite singlefile (also for EVE Asset Manager) whine here (if you prefer to be publicly ridiculed) or at me ingame (if you prefer to be silently ignored) in case there are any issues or you need custom dumps (partial sqlite, csv, ...) or something... hints on how to use the split materials data may be found in the dom100 thread. script of the changes i made compared to the ccp mssql can be found here. those are mostly changes to form, like adjusting down some integer types, removing a bunch of non-attached foreign keys, and a whole pile of FK definitions.
Lumy Minmatar eXceed Inc. HYDRA RELOADED	Posted - 2010.08.31 17:40:00 - [4] Thanks for export and conversions. There is one error I noticed in MySQL conversion. Dunno if it is error caused by conversion or it's error in original. Table: eveIcons iconID: 2998 iconFile: Black Chip (100) description: 56_16 - Unknown Joomla! in EVE - IGB compatible CMS.
SmallBrain	Posted - 2010.08.31 18:05:00 - [5] Originally by: Lumy Thanks for export and conversions. There is one error I noticed in MySQL conversion. Dunno if it is error caused by conversion or it's error in original. Table: eveIcons iconID: 2998 iconFile: Black Chip (100) description: 56_16 - Unknown Extract from the client cache files: <icons> <iconID>2998</iconID> <iconFile>Black Chip (100)</iconFile> <description>56_16 - Unknown</description> <obsolete>False</obsolete> <iconType>None</iconType> </icons> So clearly it is a problem in the original :)
SmallBrain	Posted - 2010.08.31 19:01:00 - [6] Edited by: SmallBrain on 31/08/2010 19:02:00 Edited by: SmallBrain on 31/08/2010 19:01:09 Combining both the MySQL export as the supplied database layout, I present a special PostgreSQL version: http://eve.liefdeis.com/tyr104/ It is a quoted PostgreSQL version (so invTypes doesn't work. "invTypes" does). Also, all the constraints are intact, giving strong validation of integrity of the static dump. I made this dump for my own projects, but as someone might also find use in it, I post it here. Jercy Fravowitz, thank you ever so much for making the MySQL export. Your work is very much appreciated! (and in fact, the postgres-data file is your export, minus the table creation. I hope you don't mind me 'stealing' it :) ).
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.08.31 19:17:00 - [7] Originally by: Lumy Thanks for export and conversions. There is one error I noticed in MySQL conversion. Dunno if it is error caused by conversion or it's error in original. Table: eveIcons iconID: 2998 iconFile: Black Chip (100) description: 56_16 - Unknown actualy, i have a question first: whats wrong with that row? yes, it looks mildy funneh. but afaict it is unused. if you are interested in filing bug reports about unused crud rows in the DB, i can recommend some tables ... X) Originally by: SmallBrain Extract from the client cache files: hax. Originally by: SmallBrain So clearly it is a problem in the original :) there are/were actualy some new artifacts in this dump that are closer to my definition of "error" than this particular example here. f.ex. there are a halfdozen rows in crtCertificates that reference iconIDs not in eveIcons. (referential integrity violation. reported and defected, not holding my breath on ccp content troopers actualy cleaning up something like that though.) another example i found during the "sanity checks after initial import" stage of the conversion dance there is a new row in mapDenormalize. wait, changes to the static universe as we know it? i didnt know any were scheduled! \| itemID \| itemName \| \| 61000006 \| KZFV-4 II - imtesting stuffleaveme Corp Amarr Factory Outpost \|
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.08.31 19:44:00 - [8] Originally by: SmallBrain Jercy Fravowitz, thank you ever so much for making the MySQL export. Your work is very much appreciated! (and in fact, the postgres-data file is your export, minus the table creation. I hope you don't mind me 'stealing' it :) ). dont mind at all, have fun dealing with the other postgres users. :P my mysql-to-postgres text-to-text conversion-routine is actualy publicly avail in the conversion dance script, just search for postgres there. the tools and aux files should all be avail as f.ex. http://zofu.no-ip.de/mysql2pgsql.pl, and for a quoted version one would have to remove the --noquotes option. the script also has ready-to-go command blocks for at least two csv versions and xml dumps and other conversions that i do not consider generally useful but have done at some point for someone ...
SmallBrain	Posted - 2010.08.31 20:14:00 - [9] Edited by: SmallBrain on 31/08/2010 20:14:25 Originally by: Jercy Fravowitz (..) f.ex. there are a halfdozen rows in crtCertificates that reference iconIDs not in eveIcons. (referential integrity violation. reported and defected, not holding my breath on ccp content troopers actualy cleaning up something like that though.) (..) Did you also notice there are no constraints on this iconID? :D I think they already know about it :p Found a few oddities myself regarding constraints: ALTER TABLE dbo.invTypeMaterials ADD CONSTRAINT invTypeMaterials_FK_materialType FOREIGN KEY (typeID) REFERENCES invTypes(materialTypeID) there is no invTypes.materialTypeID. ALTER TABLE invTypes ADD CONSTRAINT invTypes_FK_graphic FOREIGN KEY (graphicID) REFERENCES eveIcons(graphicID) I doubt eveIcons has a field graphicID ;) Reported both to Zirnitra, hopefully in the next dump those can be fixed :) (I hate manual work!)
Lumy Minmatar eXceed Inc. HYDRA RELOADED	Posted - 2010.08.31 22:52:00 - [10] Originally by: Jercy Fravowitz Originally by: Lumy Thanks for export and conversions. There is one error I noticed in MySQL conversion. Dunno if it is error caused by conversion or it's error in original. Table: eveIcons iconID: 2998 iconFile: Black Chip (100) description: 56_16 - Unknown actualy, i have a question first: whats wrong with that row? yes, it looks mildy funneh. but afaict it is unused. if you are interested in filing bug reports about unused crud rows in the DB, i can recommend some tables ... X) it should be Table: eveIcons iconID: 2998 iconFile: 56_16 description: Black Chip (100) - Unknown Yes, you're right it's not used by game. Yet, that's not an excuse :P Joomla! in EVE - IGB compatible CMS.
Zhou Wuwang Federal Laboratories	Posted - 2010.09.02 14:53:00 - [11] Thanks to CCP and the folks involved for providing these data dumps.
Primitive Complexity	Posted - 2010.09.18 17:36:00 - [12] Edited by: Primitive Complexity on 18/09/2010 17:41:21 Hi, Going with Jercy Fravowitz's mysql singlefile trying to import it into the database via BigDump getting the following error: Quote: Processing file: tyr104-mysql5-v1.sql Starting from line: 231001 Error at the line 233171: ) ENGINE=MyISAM DEFAULT CHARSET=utf8; Query: CREATE TABLE `eveGraphics` ( `graphicID` smallint(6) NOT NULL default '0', `graphicFile` varchar(500) NOT NULL, `description` varchar(16000) NOT NULL, `obsolete` tinyint(1) NOT NULL, `graphicType` varchar(100) default NULL, `collidable` tinyint(1) default NULL, `explosionID` smallint(6) default NULL, `directoryID` int(11) default NULL, `graphicName` varchar(64) NOT NULL, PRIMARY KEY (`graphicID`), KEY `explosionID` (`explosionID`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8; MySQL: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs Haven't ran across that one before, any clues? Manually looking through the INSERTS into eveGraphics I don't see anything that should/could exceed 65535 characters.. At any rate, Jercy, big thanks for all of the quick conversions with these dumps :) EDIT: Just in case its important, im using PHP 6.0.0-dev and MySQL 6.0.4-alpha
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.09.18 19:52:00 - [13] Originally by: Primitive Complexity MySQL 6.0.4-alpha That explains it, I guess. Had no issues loading it to 5,ч (Both local and ISP DB) -- Thanks CCP for cu
Primitive Complexity	Posted - 2010.09.18 21:53:00 - [14] Originally by: Tonto Auri Originally by: Primitive Complexity MySQL 6.0.4-alpha That explains it, I guess. Had no issues loading it to 5,ч (Both local and ISP DB) Confirming that forementioned issue does not occur with MySQL 5.0.51b. Thanks for the tip Tonto,
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.09.21 13:34:00 - [15] Originally by: Jercy Fravowitz conversions: mysql singlefile (you are most likely looking for this one) this one should also load just fine in php-webloader-scripts, if it whines about something with timestamps at the end, ignore it. sqlite singlefile (also for EVE Asset Manager) whine here (if you prefer to be publicly ridiculed) or at me ingame (if you prefer to be silently ignored) in case there are any issues or you need custom dumps (partial sqlite, csv, ...) or something... hints on how to use the split materials data may be found in the dom100 thread. script of the changes i made compared to the ccp mssql can be found here. those are mostly changes to form, like adjusting down some integer types, removing a bunch of non-attached foreign keys, and a whole pile of FK definitions. Ok, whining... Tried to import SQL dump to MySQL/InnoDB. Better said, "trying". Started at: 7:27:35 Uptime: 10:05:12 It's INCREDIBLE slow... anyone have clever idea, what's going on? And how to increase speed? -- Thanks CCP for cu
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.09.21 18:16:00 - [16] Ok, I have it... Quote: $ ./import.sh tyr104-mysql5-v1-NE-AC.sql real 24m18.109s $ ./import.sh tyr104-mysql5-v1-MyISAM-EI-NA.sql real 4m31.278s $ ./import.sh tyr104-mysql5-v1-InnoDB-NE-NA.sql real 51m34.231s $ ./import.sh tyr104-mysql5-v1-InnoDB-EI-NA.sql real 14m38.973s NE - No Extended Inserts AC - AutoCommit NA - No AutoCommit EI - Extended Inserts (multiple values merged in a row) MyISAM AC NE (original dump) - 25min. MyISAM NA NE ("SET AUTOCOMMIT=0;" before and "COMMIT;" every 300-500 INSERT's) - 15min. (not shown there) MyISAM NA EI - incredible 5 min. InnoDB NA EI - Very affrodable 15min. InnoDB NA NE - An hour. Poor, but bearable. InnoDB AC EI - More than 24h (estimated). InnoDB AC NE - More than two days (again, estimated for about 2.5d). -- Thanks CCP for cu
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.09.22 16:37:00 - [17] Real cluster experience: Quote: $ time bzcat tyr104-mysql5-v1-InnoDB-EI-NA.bz2 \| mysql --host=mysql... --user=... -B -p -- ...db 2>&1> import.log Enter password: real 9m48.671s user 0m27.425s sys 0m2.436s Now, I'm officially happy. -- Thanks CCP for cu
Philderbeast	Posted - 2010.11.03 10:42:00 - [18] are there any major changes with the new inventory system introduced in 1.2? aka do i need to get a new dump to continue developing my app?
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.11.03 20:12:00 - [19] Originally by: Philderbeast are there any major changes with the new inventory system introduced in 1.2? aka do i need to get a new dump to continue developing my app? There was no changes in items, ships, stations, agents... Nada. The last dump is still the actual one. -- Thanks CCP for cu
Sidrat Flush Caldari Eve Industrial Corp	Posted - 2010.11.04 16:29:00 - [20] I wonder if CCP could actually produce their own buildtypereq table into the datadump? In the meantime if fubar could update the sql code for his marvellous efforts that would be great. View The Eve Industrial Organiser Site
Magic Crisp Minmatar	Posted - 2010.11.04 17:57:00 - [21] I'd like to ask something, here[1] it's told that the typeID field of ramTypeRequirements should be a reference to invBlueprintTypes.blueprintTypeID, that seems to make sense, we're speaking of blueprint requirements. However, I've defined an FK on this and it's violated at population. After this I've run the following query in the MSSQL database: SELECT distinct [typeID] FROM [tyrannis].[dbo].[ramTypeRequirements] WHERE NOT typeID IN (SELECT blueprintTypeID FROM [tyrannis].[dbo].[invBlueprintTypes]) And i've got a result of 359 rows. These are the items in ramTypeRequirements.typeID that are _not_ listed in the blueprint table. Checked the results, they are really not blueprints. May I ask what should ramTypeRequirements.typeID be a reference for? how should i define an FK to keep consitency? Thanks in advance. [1] http://wiki.eve-id.net/RamTypeRequirements_(CCP_DB)
Dragonaire Caldari Corax. SOUL CARTEL	Posted - 2010.11.05 03:12:00 - [22] Since you shouldn't be making changes to any of the DB dump tables but only use them read only why would you need FK to start with? Also remember that many of the tables in the data dump are extracted or merges etc from the actual tables that CCP uses so they don't always have any true consistency between them. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
Magic Crisp Minmatar	Posted - 2010.11.05 07:59:00 - [23] I'm putting the FK on my 1:1 conversion table, which is in PostgreSQL. I'm reading the mssql DB with a .net app, and inserting the data into postgresql. I'm trying to define the FKs to have a proper DB at the end, but this one is bugging me.
Lutz Major Austriae Est Imperare Orbi Universo	Posted - 2010.11.05 08:41:00 - [24] Originally by: Magic Crisp I'm putting the FK on my 1:1 conversion table, which is in PostgreSQL. I'm reading the mssql DB with a .net app, and inserting the data into postgresql. I'm trying to define the FKs to have a proper DB at the end, but this one is bugging me. Well then good luck. The ramTypeRequirements table is not the only one that misses referential integrity. IMHO you are wasting your time. The dump is all in all consistent () and if some tables lack integrity I don't care. I find it wonderful, that CCP provides such stuff \o/
Darlington VIRTUAL LIFE VANGUARD	Posted - 2010.11.05 20:49:00 - [25] Hi All Is there anywhere I can get a download of specific tables? There are only a few I need Regards D
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.11.06 00:59:00 - [26] Originally by: Darlington Hi All Is there anywhere I can get a download of specific tables? There are only a few I need Download a complete dump and strip it as you wish. I don't understand what you intend by asking for "download of specific tables". Originally by: Darlington Regards D There's a forum setting called "signature" to automate insertion of such things. -- Thanks CCP for cu
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.11.06 17:21:00 - [27] Originally by: Magic Crisp May I ask what should ramTypeRequirements.typeID be a reference for? how should i define an FK to keep consitency? from my list of FKs applied during the sanity-check phase of the conversion dance: ALTER TABLE `ramTypeRequirements` ADD FOREIGN KEY (`typeID`) REFERENCES `invTypes` (`typeID`); ALTER TABLE `ramTypeRequirements` ADD FOREIGN KEY (`requiredTypeID`) REFERENCES `invTypes` (`typeID`); ALTER TABLE `ramTypeRequirements` ADD FOREIGN KEY (`activityID`) REFERENCES `ramActivities` (`activityID`); there are activities other than "build from blueprint" that have requirements. so there are typeIDs in there that are not blueprints. there is one block of activityID=1 rows in there for a typeid that is not a blueprint though. if you look at the types, it seems like someone confused product and blueprint. if you want to br something, br those six rows for typeID=23418.
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.11.06 17:31:00 - [28] Originally by: Tonto Auri real 9m48.671s Now, I'm officially happy. sometimes i am wondering why people seem to be running their databases on leftover cellphones from 1992. the move to no-extended-inserts was a while back and completely eliminated the requests for separate dumps that would load in some "load database into webhost" scripts. why anyone would write a mysql import script that is unable to handle the mysqldump default format is beyond me. well, these things are written in php ... anyways, ran numbers on it, and it looked like the right thing to do. uncompressed size increases 30-40% or so, but compressed size stays the same. loading time ... changed from 1 minute to 2 minutes here. yes, twice as long, but still just a minute more. who cares. the released dump is basicly a -NE-AC > bzcat tyr104-mysql5-v1.sql.bz2 \| time mysql $MYSQLOPT $DBNAME 16.10user 8.28system 1:51.21elapsed 21%CPU thats 1 minute 51 seconds for the plain import. > bzcat tyr104-mysql5-v1.sql.bz2 \| sed -e 's/ENGINE=MyISAM/ENGINE=InnoDB/g' \| time mysql $MYSQLOPT $DBNAME 16.66user 8.57system 7:50.36elapsed 5%CPU 7 minute 50 sec to load it with innodb. > mysqldump --extended-insert --no-autocommit $MYSQLOPT $DBNAME > tyr104-mysql5-v1-EI-NA.sql 51 sec myisam, 7m50s innodb. note the innodb value basicly didnt change at all. myisam shows the "expected" one minute difference the extended inserts make. > mysqldump --skip-extended-insert --no-autocommit $MYSQLOPT $DBNAME > tyr104-mysql5-v1-NE-NA.sql 1m42s myisam, 7m33s innodb. so, no matter what, the innodb value changes little to none for me. myisam shows the expected impact of the extended inserts. Tonto: can you please try to load this one into your innodb? tyr104-mysql5-v1-NE-NA.sql.bz2 if that one has bearable performance for you, i will add NA to future dumps.
Jercy Fravowitz School of Applied Knowledge	Posted - 2010.11.06 17:47:00 - [29] Originally by: Darlington Is there anywhere I can get a download of specific tables? here
Tonto Auri Vhero' Multipurpose Corp	Posted - 2010.11.07 02:52:00 - [30] Originally by: Jercy Fravowitz Originally by: Tonto Auri real 9m48.671s Now, I'm officially happy. sometimes i am wondering why people seem to be running their databases on leftover cellphones from 1992. Because it's what I have? It's not like I'm going to change my development box just because a year have passed. (And it's from 2000... originally, the meainboard died since and was replaced in 2007, other than that - still the same Athlon 900 @ VIA 8363A/686B) The target (production) platform is indeed better. Hence the minimal difference from your own results. Quote: the move to no-extended-inserts was a while back and completely eliminated the requests for separate dumps that would load in some "load database into webhost" scripts. I know. Besides, I didn't complained, nor requested anything, except some assistance. Quote: why anyone would write a mysql import script that is unable to handle the mysqldump default format is beyond me. well, these things are written in php ... That's a legacy of the webhosting childhood. Today, any reasonable hosting company offer a shell access. You could find offers as cheap as $30/year including shell/PHP/MySQL/domain name with mailing support. Quote: Tonto: can you please try to load this one into your innodb? tyr104-mysql5-v1-NE-NA.sql.bz2 if that one has bearable performance for you, i will add NA to future dumps. I'll try it in a day or two. -- Thanks CCP for cu

Pages: [1] 2 :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.