Pages: 1 2 3 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Desmont McCallock
569
|
Posted - 2015.11.15 13:09:42 -
[1] - Quote
Introducing the next generation of EVE SDE handling tool (formely known as EVEMonSDEExternalsToSql).
I'm presenting to you EVESDEToSQL v3. Version 3 is the evolution of the previous version, containing all the goodies of v2, plus the ability to export the EVE SDE to the most popular SQL formats and more.
The tool source code has been separated from EVEMon and now resides in its own repository.
The compiled version can be found for downloading here.
The latest version is: 3.0.0
Direct dl link: EVESDEToSQL-3.0.0.zip
Notice: The SQL data dump, yaml and SQLite db files are not distributed along with the tool. You will have to get them from the community toolkit page.
--Features--
-Importation
* Restores the data dump, yaml and sqlite db files back into an SQL Server (no need to restore the data dump via SSMS). * Imports any SDE (literally any, but in case you find an SDE with issues please let me known) * Supports command line arguments (use 'help' to see the list) * Cleaner config file (it now only contains the needed connection info to your SQL Server, and it's the only thing that you may need to modify) * Ulta-High Speed (it now takes less than 5 minutes to import the entire SDE, depending on your machine's cpu power) * One place for the SDE file(s) (just drop the SDE zip file as is or the files contained in the SDE zip file, into the 'SDEFiles' folder)
-Exportation
* Exports an SQL Server DATADUMP backup file * Exports an SQLite database file * Exports an MS Access database file ('mdb' as default, use '/ace' switch for 'accdb') * Exports an MySQL dump file * Exports an PostgreSQL dump file * Exports individual CSV (semicolon seperated version) files * Ability to export specific tables in any dump format
Supported OS: Windows XP, Vista, 7, 8, 8.1, 10
Exportation Usage Examples
SQL Server dump: EVESDEToSQL export -sql [/et: listOfTables] MySQL dump: EVESDEToSQL export -mysql [/et: listOfTables] PostgreSQL dump: EVESDEToSQL export -postgresql [/et: listOfTables] SQLite dump: EVESDEToSQL export -sqlite [/et: listOfTables] Ms Access dump: EVESDEToSQL export -access [/ace] [/et: listOfTables] CSV dump: EVESDEToSQL export -csv [/et: listOfTables]
Requirements
* .NET 4.0 or greater * SQL Server instance (any version, even Express)
Yaml files mapping:
blueprints.yaml -> invBlueprintTypes, ramTypeRequirements categoryIDs.yaml -> invCategories, translationTables, trnTranslationColumns, trnTranslations certificates.yaml -> crtClasses, crtCertificates, crtRecommendations, crtRelationships graphicIDs.yaml -> eveGraphics groupIDs.yaml -> invGroups, translationTables, trnTranslationColumns, trnTranslations iconIDs.yaml -> eveIcons skinLicenses.yaml -> sknLicenses skinMaterials.yaml -> sknMaterials skins.yaml -> sknSkins tournamentRuleSets.yaml -> ignored typeIDs.yaml -> invTypes, dgmMasteries, dgmTypeMasteries, dgmTraits, dgmTypeTraits, translationTables, trnTranslationColumns, trnTranslations
Sqlite files mapping:
universeDataDx.db -> mapCelestialStatistics, mapConstellationJumps, mapConstellations, mapDenormalize, mapJumps, mapLandmarks, mapLocationScenes, mapLocationWormholeClasses, mapRegionJumps, mapRegions, mapSolarSystemJumps, mapSolarSystems
Known Issues:
None (so far)
Enjoy responsibly. |
Desmont McCallock
569
|
Posted - 2015.11.20 09:27:45 -
[2] - Quote
FAQ
Q:I don't want the name of the database to be 'EVEStaticData'.
A: Edit the EVESDEToSQL.exe.config file and change the value of connection's string 'Initial Catalog' to name the database differently.
Q:I'm getting the error message Connection to SQL Server 'EveStaticData' database: Failed Reason was: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
A: Edit the EVESDEToSQL.exe.config file and adjust the value of connection's string 'Data Sourse' to point to your SQL Server instance (the tool assumes that the SQL Server instance is named SQLEXPRESS).
Q:I'm getting the error message Connection to SQL Server 'EveStaticData' database: Failed Reason was: Cannot open database "EveStaticData" requested by the login. The login failed. Login failed for user 'computerName\user'.
A: This message appears when you try to export to a dump format. Do an import first (EVESDEToSQL import). |
Desmont McCallock
569
|
Posted - 2015.11.20 10:56:45 -
[3] - Quote
Releasing version 3.0.1.
- Includes several bug fixes and improved error handling. |
Golden Gnu
EVE University Ivy League
174
|
Posted - 2015.11.23 14:21:23 -
[4] - Quote
Thanks Desmont! :) I'm still only using the import feature, but, it's amazing how this software always works without any problem. You must have a talent for writing high quality code o7
Download is the meaning of life, upload is the meaning of intelligent life
http://eve.nikr.net - home of jEveAssets
|
Desmont McCallock
572
|
Posted - 2015.11.23 16:48:34 -
[5] - Quote
Golden Gnu wrote:Thanks Desmont! :) I'm still only using the import feature, but, it's amazing how this software always works without any problem. You must have a talent for writing high quality code o7
Thank you for the compliment.
|
GrayLensman
Galactic Protectorate
0
|
Posted - 2015.12.01 01:52:41 -
[6] - Quote
Desmont,
I have attempted to run your utility several times but am clearly missing something. I have attached two images:
(1) showing the location of EVESDEToSQL and the contents of the SDEFiles folder first...
Picture 1
(2) Then showing the results and that EVESDEToSQL is not able to locate the SDE files...
Picture 2
What am I missing? Thanks.
Gray Lensman |
Desmont McCallock
574
|
Posted - 2015.12.01 08:10:51 -
[7] - Quote
It looks like the unzipping is failing. I'll have a look at it. Meanwhile don't use the zip file inside the folder but just unzip the files in it and remove the zip file afterwards. |
Desmont McCallock
574
|
Posted - 2015.12.01 08:37:31 -
[8] - Quote
Released version 3.1.0.
Change log * Added support for using SQL Server Authentication. * Fixed a bug when trying to unzip the SDE zip file. |
GrayLensman
Galactic Protectorate
0
|
Posted - 2015.12.01 16:04:51 -
[9] - Quote
3.1 tested and confirmed, results of import below provided in 57.38 seconds... this really is excellent work, nice job...
Considering the amount of labor you saved me I have contributed 50m isk to the Desmont McCallock Software Services Fund... I encourage other pilots to do the same.
D:\Dev\Data\EVESDEToSQL-3.1.0>evesdetosql import Unzipping SDE zip file: Successful
Importing SQL data dump to 'EveStaticData' database... 100 % in 0:00:02.2645004
Connection to SQL Server 'EveStaticData' database: Successful
Importing categoryIDs.yaml... 100 % in 0:00:00.1473313 Importing groupIDs.yaml... 100 % in 0:00:00.2944629 Importing graphicIDs.yaml... 100 % in 0:00:00.3124175 Importing iconIDs.yaml... 100 % in 0:00:00.1020952 Importing skins.yaml... 100 % in 0:00:00.0881721 Importing skinMaterials.yaml... 100 % in 0:00:00.0217246 Importing skinLicenses.yaml... 100 % in 0:00:00.0708117 Importing typeIDs.yaml... 100 % in 0:00:16.3830429 Importing certificates.yaml... 100 % in 0:00:00.2589833 Importing blueprints.yaml... 100 % in 0:00:02.8515145
Connection to SQLite 'main' database: Successful
Importing mapCelestialStatistics... 100 % in 0:00:09.2347243 Importing mapConstellationJumps... 100 % in 0:00:00.0475246 Importing mapConstellations... 100 % in 0:00:00.0408733 Importing mapDenormalize... 100 % in 0:00:19.1871954 Importing mapJumps... 100 % in 0:00:00.0954077 Importing mapLandmarks... 100 % in 0:00:00.0177922 Importing mapLocationScenes... 100 % in 0:00:00.0150901 Importing mapLocationWormholeClasses... 100 % in 0:00:00.0198079 Importing mapRegionJumps... 100 % in 0:00:00.0171859 Importing mapRegions... 100 % in 0:00:00.0250435 Importing mapSolarSystemJumps... 100 % in 0:00:00.22791 Importing mapSolarSystems... 100 % in 0:00:00.2612589
Disconnection from SQLite 'main' database: Successful
Reorganizing indexes of database 'EveStaticData'... 100 % in 0:00:05.085969
Shrinking database 'EveStaticData'... in 0:00:00.1521892
Disconnection from SQL Server 'EveStaticData' database: Successful
Importing files completed in 0:00:57.3862157
Press any key to exit. D:\Dev\Data\EVESDEToSQL-3.1.0> |
Desmont McCallock
574
|
Posted - 2015.12.01 19:23:50 -
[10] - Quote
@GrayLensman Very much appreciated. |
|
Desmont McCallock
585
|
Posted - 2015.12.07 10:23:25 -
[11] - Quote
For those having issue with using the SDE zip file, make sure that the file is named according to CCP's naming convention 'ReleaseName_VersionMajor.VersionMinor_VersionBuild_db.zip' (which they don't seem to honor in some cases). |
Hel O'Ween
Men On A Mission
153
|
Posted - 2015.12.10 09:51:28 -
[12] - Quote
Continuing from here
Running the tool with ...
EVESDEToSQL import -norestore -noyaml
... still fails with the already mentioned timeout and produces the below content in trace.txt:
Quote: EVESDEToSQL: Started Native library pre-loader is trying to load native SQLite library "C:\DATEN\x86\SQLite.Interop.dll"... Deleting file C:\DATEN\SDEFiles\universeDataDx.db System.IO.IOException: Der Prozess kann nicht auf die Datei "C:\DATEN\SDEFiles\universeDataDx.db" zugreifen, da sie von einem anderen Prozess verwendet wird. bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) bei System.IO.File.InternalDelete(String path, Boolean checkHost) bei System.IO.File.Delete(String path) bei EVESDEToSQL.Utils.Util.DeleteSDEFilesIfZipExists() in EVESDEToSQL\Utils\Util.cs:Zeile 184.
Der Prozess kann nicht auf die Datei "C:\DATEN\SDEFiles\universeDataDx.db" zugreifen, da sie von einem anderen Prozess verwendet wird.
translates to
The process can't access file "C:\DATEN\SDEFiles\universeDataDx.db", becuase it is in use by another process.
I peeked at Util.cs and it seems it tries to delete the SQLite DB while it's still in use somewhere. My C# is too weak to determine the real cause, but I suspect this is a follow-up error of the import issue, which leaves the SQLite DB still connected/open, and this deletion exception happens during the "normal" cleanup operation when closing the application.
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
586
|
Posted - 2015.12.10 13:42:59 -
[13] - Quote
Hel O'Ween wrote:I peeked at Util.cs and it seems it tries to delete the SQLite DB while it's still in use somewhere. My C# is too weak to determine the real cause, but I suspect this is a follow-up error of the import issue, which leaves the SQLite DB still connected/open, and this deletion exception happens during the "normal" cleanup operation when closing the application. This is a normal error as the tool tries to delete the files it had unzipped once you close it. The fact that it's still failing to import only the sqilte DB is troublesome. Just for the record, I can't reproduce it on my machine but I'll investigate this behavior further.
|
Hel O'Ween
Men On A Mission
153
|
Posted - 2015.12.10 18:06:20 -
[14] - Quote
Desmont McCallock wrote: The fact that it's still failing to import only the sqilte DB is troublesome. Just for the record, I can't reproduce it on my machine but I'll investigate this behavior further.
It does import the sqlite quite find, it just chokes on that single mapDenormalize table after the counter goes up to 100% for it.
I just dropped mapDenormalize from the sqlite and tried the import again: worked like a charm. Yeah, mapDenomralize is the biggest table, but it's still "peanuts" for any really RDBMS.
Thanks, but the good ol' EVEMonSDEExternalsToSql.exe already did the job for me.
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
589
|
Posted - 2015.12.19 10:23:34 -
[15] - Quote
Iterating again on your problem here is what I found so far:
Fact: Memory consumption is maximum 500MB during the entire operation. Conclusion: Therefore it's mot a memory consumption issue. Action: No action will be taken.
Fact: The bulk insert timeout is fixed at 30 seconds and is not linked to the connection timeout setting in the connection string. Conclusion: Most likely the problem lies there, as from looking at the times it takes on your machine to insert the other tables, MapDenormalize exceeds the timeout and any offset it might have. Action: I will add some code to link the connection timeout set in the connection string to the bulk insert timeout. |
Desmont McCallock
589
|
Posted - 2015.12.19 10:32:15 -
[16] - Quote
Released version 3.1.1.
Change Log * Improved search algorithm for the SDE zip file. * Linked the bulk insert operation timeout to the connection timeout if it's set and exceeds the default value. |
Desmont McCallock
589
|
Posted - 2015.12.19 10:39:26 -
[17] - Quote
@Hel O'Ween Try again setting 'Connection Timeout=60' in the connection string. |
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.19 10:51:00 -
[18] - Quote
I'm not having access to my dev machine right now, will try that on monday.
Regardless of the outcome - many thanks for your ongoing efforts. Much appreciated, Desmont!
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
590
|
Posted - 2015.12.19 11:20:30 -
[19] - Quote
Hel O'Ween wrote:I'm not having access to my dev machine right now, will try that on monday.
Regardless of the outcome - many thanks for your ongoing efforts. Much appreciated, Desmont! Sure thing. Thanks for your kind words. |
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.21 12:12:04 -
[20] - Quote
And surely enough, that fixed it.
Thanks, Desmont.
EVEWalletAware - an offline wallet manager.
|
|
Desmont McCallock
591
|
Posted - 2015.12.21 12:29:49 -
[21] - Quote
Hel O'Ween wrote:And surely enough, that fixed it. Thanks, Desmont. Thanks for confirming.
|
salacious necrosis
Federal Defense Union Gallente Federation
14
|
Posted - 2015.12.24 18:14:27 -
[22] - Quote
Hi Des,
Used your exporter for the first time today. Everything worked well except for SQLite import. To get that to work I had to copy x86/SQLite.Interop.dll to the main EVESDEToSQL directory. Otherwise, the importer fails with a trace.txt like this:
Quote: EVESDEToSQL: Started Native library pre-loader is trying to load native SQLite library "C:\Users\mark_000\projects\sdeimport\EVESDEToSQL\x86\SQLite.Interop.dll"... Native library pre-loader is trying to load native SQLite library "C:\Users\mark_000\projects\sdeimport\EVESDEToSQL\x86\SQLite.Interop.dll"... System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.DllNotFoundException: Unable to load DLL 'SQLite.Interop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_none(SQLiteConfigOpsEnum op) at System.Data.SQLite.SQLite3.StaticIsInitialized() at System.Data.SQLite.SQLiteLog.Initialize() at System.Data.SQLite.SQLiteConnection..ctor(String connectionString, Boolean parseViaFramework) at System.Data.SQLite.SQLiteConnection..ctor(String connectionString) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at EVESDEToSQL.Providers.DbConnectionProvider.CreateConnection(Type connectionType, String nameOrConnectionString) in EVESDEToSQL\Providers\DbConnectionProvider.cs:line 97 at EVESDEToSQL.Providers.SqliteConnectionProvider..ctor(String nameOrConnectionString) in EVESDEToSQL\Providers\SqliteConnectionProvider.cs:line 22 at EVESDEToSQL.Importer.ImportSqlite() in EVESDEToSQL\Importer.cs:line 185 at EVESDEToSQL.Importer.ImportSDEFiles(String[] args) in EVESDEToSQL\Importer.cs:line 135 at EVESDEToSQL.Importer.Start(String[] args) in EVESDEToSQL\Importer.cs:line 90
System.DllNotFoundException: Unable to load DLL 'SQLite.Interop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_none(SQLiteConfigOpsEnum op) at System.Data.SQLite.SQLite3.StaticIsInitialized() at System.Data.SQLite.SQLiteLog.Initialize() at System.Data.SQLite.SQLiteConnection..ctor(String connectionString, Boolean parseViaFramework) at System.Data.SQLite.SQLiteConnection..ctor(String connectionString)
My box is x64 but it didn't try to load from that directory at all. Anyway, happy to help debug further if you provide instructions.
|
Desmont McCallock
591
|
Posted - 2015.12.25 13:13:10 -
[23] - Quote
Although I suspect it's mostly a SQLite issue I'll surely look into it. |
Desmont McCallock
591
|
Posted - 2015.12.25 13:14:22 -
[24] - Quote
Is there any interest in providing an SQL CE (SQL Server Compact 4) dump? |
salacious necrosis
Federal Defense Union Gallente Federation
14
|
Posted - 2015.12.25 18:34:07 -
[25] - Quote
Desmont McCallock wrote:Although I suspect it's mostly an SQLite issue I'll surely look into it.
It doesn't matter if your box is x64. The app is set to run as x86 (to provide support fro MS Access mdb files).
Never the less, I would much appreciate if you could tell me what OS your box is running on.
Running Windows 8.1 Pro, with .NET 4.0.30319
I don't have any of the SQLite dll's in my path except for the those included in EVESDEToSQL. I DO have the sqlite3 CLI binary in my path, but I don't think that should affect anything.
|
Desmont McCallock
591
|
Posted - 2015.12.26 14:08:12 -
[26] - Quote
So I checked the tool again and I can't find anything wrong with it.
The only way I could reproduce the exact exception you are reporting, is to rename the 'x64' folder to 'x86', to fool the app so it tries to load the 64bit version of the interop library.
So for some reason on your machine the tool although it says it's looking into 'x86' it tries to load the 64bit dll or for some unknown reason to me the interop dlls got mixed up.
I suggest you delete the folder where the tool currently resides and re-download it.
Just for the reference here is the Dynamic-Link Library Search Order that Windows uses.
- The directory from which the application loaded.
- The system directory. Use the GetSystemDirectory function to get the path of this directory.
- The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched.
- The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
- The current directory.
- The directories that are listed in the PATH environment variable. Note that this does not include the per-application path specified by the App Paths registry key.
|
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.28 11:10:30 -
[27] - Quote
I'm very new to this .NET stuff, so I might be totally of here, but I remember something about assemblies being registered in the GAC ... might something like this cause the lookup order to not working as expected?
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
591
|
Posted - 2015.12.28 12:41:42 -
[28] - Quote
Hel O'Ween wrote:I'm very new to this .NET stuff, so I might be totally of here, but I remember something about assemblies being registered in the GAC ... might something like this cause the lookup order to not working as expected? You are correct Hel O'Ween. It's a possibility.
|
Desmont McCallock
598
|
Posted - 2016.02.05 11:46:08 -
[29] - Quote
Released version 3.2.0.
ChangeLog Updated SQLite nuget package. Updated YamlDotNet nuget package. Included missing SMO assembly (was though available in GAC if SQL Server was installed on the machine). |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.01 18:14:26 -
[30] - Quote
Hi Desmont,
This tool seems really useful to me! However i've run into a bit of an issue.
Reason was: System.Data.SqlClient.SqlError: Cannot open backup device 'C:\Users\ Simon\Downloads\EVESDEToSQL-3.2.0\SDEFiles\DATADUMP201602091443.bak'. Operating system error 3(The system cannot find the path specified.).
Press any key to exit.
Any idea's to what I've done wrong? haha |
|
Desmont McCallock
599
|
Posted - 2016.03.01 20:11:10 -
[31] - Quote
Grenval Truvaki wrote:Hi Desmont,
This tool seems really useful to me! However i've run into a bit of an issue.
Reason was: System.Data.SqlClient.SqlError: Cannot open backup device 'C:\Users\ Simon\Downloads\EVESDEToSQL-3.2.0\SDEFiles\DATADUMP201602091443.bak'. Operating system error 3(The system cannot find the path specified.).
Press any key to exit.
Any idea's to what I've done wrong? haha Have you followed instructions from OP? See also https://bitbucket.org/Desmont_McCallock/evesdetosql |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.01 21:18:09 -
[32] - Quote
Desmont McCallock wrote:Grenval Truvaki wrote:Hi Desmont,
This tool seems really useful to me! However i've run into a bit of an issue.
Reason was: System.Data.SqlClient.SqlError: Cannot open backup device 'C:\Users\ Simon\Downloads\EVESDEToSQL-3.2.0\SDEFiles\DATADUMP201602091443.bak'. Operating system error 3(The system cannot find the path specified.).
Press any key to exit.
Any idea's to what I've done wrong? haha Have you followed instructions from OP? See also https://bitbucket.org/Desmont_McCallock/evesdetosql
Yes I have, however I think my database is missing some tables now. I'm doing a fresh run of the SDE on my MS SQL server.
Right now i've managed to get this far:
C:\Users\Simon>\Downloads\EVESDEToSQL-3.2.0\EVEsdetosql import -no restore Unzipping SDE zip file: Successful
Connection to SQL Server 'DB_9F457E_projectDB' database: Successful
Parsing categoryIDs.yaml... Unable to execute SQL command: UPDATE translationTables SET tcGroupID = 4, tcID = 6, sourceTable = 'inventory.categoriesTx', destinationTable = 'dbo.invCategori es', translatedKey = 'categoryName' WHERE sourceTable = 'inventory.categoriesTx' AND translatedKey = 'categoryName' Invalid object name 'translationTables'.
Press any key to exit. |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.01 22:29:12 -
[33] - Quote
I should also mention when I try to apply the bak file on my server I'm given this error in response:
Error: The media family on device 'c:\db_temp\datadump201602091443.bak' is incorrectly formed. SQL Server cannot process this media family. |
Desmont McCallock
599
|
Posted - 2016.03.02 08:38:20 -
[34] - Quote
Grenval Truvaki wrote:I should also mention when I try to apply the bak file on my server I'm given this error in response:
Error: The media family on device 'c:\db_temp\datadump201602091443.bak' is incorrectly formed. SQL Server cannot process this media family. Ahh yes. CCP said that the SDE MSSQL backup file is created with SQL 2014. SQL instances prior to v2014 will not be able to restore the backup file.
Edit: I updated the OP requirements so it's clearer. |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.02 13:45:43 -
[35] - Quote
Desmont McCallock wrote:Grenval Truvaki wrote:I should also mention when I try to apply the bak file on my server I'm given this error in response:
Error: The media family on device 'c:\db_temp\datadump201602091443.bak' is incorrectly formed. SQL Server cannot process this media family. Ahh yes. CCP said that the SDE MSSQL backup file is created with SQL 2014. SQL instances prior to v2014 will not be able to restore the backup file. Edit: I updated the OP requirements so it's clearer.
Thank you for the help :) Don't suppose you know of any way around this? I'm stuck using MS SQL 2012. |
Desmont McCallock
599
|
Posted - 2016.03.02 16:15:11 -
[36] - Quote
Set the connection timeout in the connectionstring in the config ( http://www.connectionstrings.com/all-sql-server-connection-string-keywords/ ). |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.02 16:28:55 -
[37] - Quote
I gave a two minute timeout on the connectionstring, it's still hanging at 50% when trying to import typeIDs.yaml |
Desmont McCallock
599
|
Posted - 2016.03.03 12:10:03 -
[38] - Quote
Grenval Truvaki wrote:I gave a two minute timeout on the connectionstring, it's still hanging at 50% when trying to import typeIDs.yaml UPDATE: So after giving the string a whole 20 mins before it would timeout :P it's finally done, thank you very much for your help and for this tool! From what I understood your SQL Server is cloud based. So depending on your upload speed you had to adjust the connection timeout accordingly. |
m Light
I N E X T R E M I S Tactical Narcotics Team
0
|
Posted - 2016.03.16 22:39:28 -
[39] - Quote
o7 ty
Importing SQL data dump to 'EveStaticData' database... 100 % in 0:00:44.285446
Connection to SQL Server 'EveStaticData' database: Successful
Importing categoryIDs.yaml... 100 % in 0:00:00.4530976 Importing groupIDs.yaml... 100 % in 0:00:00.356952 Importing graphicIDs.yaml... 100 % in 0:00:00.3333244 Importing iconIDs.yaml... 100 % in 0:00:00.1151018 Importing skins.yaml... 100 % in 0:00:00.1637418 Importing skinMaterials.yaml... 100 % in 0:00:00.2591509 Importing skinLicenses.yaml... 100 % in 0:00:05.6676665 Importing typeIDs.yaml... 100 % in 0:00:44.0759559 Importing certificates.yaml... 100 % in 0:00:00.73331 Importing blueprints.yaml... 100 % in 0:00:05.4793214
Connection to SQLite 'main' database: Successful
Importing mapCelestialStatistics... 100 % in 0:00:34.3261924 Importing mapConstellationJumps... 100 % in 0:00:00.0502679 Importing mapConstellations... 100 % in 0:00:00.048732 Importing mapDenormalize... 100 % in 0:02:07.2610279 Importing mapJumps... 100 % in 0:00:00.1278917 Importing mapLandmarks... 100 % in 0:00:00.0177527 Importing mapLocationScenes... 100 % in 0:00:00.0160912 Importing mapLocationWormholeClasses... 100 % in 0:00:00.0200616 Importing mapRegionJumps... 100 % in 0:00:00.0173105 Importing mapRegions... 100 % in 0:00:00.0236341 Importing mapSolarSystemJumps... 100 % in 0:00:00.2487416 Importing mapSolarSystems... 100 % in 0:00:00.3020079
Disconnection from SQLite 'main' database: Successful
Reorganizing indexes of database 'EveStaticData'... 100 % in 0:00:36.4269115
Shrinking database 'EveStaticData'... in 0:00:13.9365718
Disconnection from SQL Server 'EveStaticData' database: Successful
Importing files completed in 0:05:15.5000529
100% == 100 mil ISK ty |
Desmont McCallock
599
|
Posted - 2016.03.17 08:58:42 -
[40] - Quote
Glad you found it useful. Thank you. |
|
Tristan Agion
Viziam Amarr Empire
134
|
Posted - 2016.04.17 17:43:58 -
[41] - Quote
Hi there!
Thanks for this excellent tool! First, let me say that everything is working fine for me as far as program setup and running is concerned. But I have a problem with some data I get from this.
Using YC-118-3_1.0_117575.zip with "import" followed by "export -csv", I get a bunch of CSV files. Among them is
mapSolarSystems.csv
in which I can find the following systems
10000004;20000047;30000327;D95-FQ; 10000004;20000047;30000328;ZSPJ-K;
where I have just copied the first four columns
regionID;constellationID;solarSystemID;solarSystemName;
Now, if we look at Dotlan, then we see that they are connected by a stargate
Dotlan map
I would expect to see this connection in
mapSolarSystemJumps.csv
Basically, there should be an entry
10000004;20000047;30000327;30000328;20000047;10000004;
which is just a connection from D95-FQ to ZSPJ-K according to
fromRegionID;fromConstellationID;fromSolarSystemID;toSolarSystemID;toConstellationID;toRegionID;
But in fact neither systems is listed here at all (the relevant SolarSystemIDs are not in this CSV file).
I am relatively sure that I'm reading the files right in general, since I get other connections right.
I can see three possibilities:
1. My understanding is false/incomplete, this particular connection is listed elsewhere or in a different manner. 2. CCP's SDE is corrupt concerning this, and Dotlan somehow knows better. 3. The SDE is correct, but something goes wrong in the import or export to CSV with your program.
I was hoping that you could shed some light on what is going on there. I should mention that I have basically zero clue about SQL databases, I just followed the various instructions to set stuff up (apparently successfully, everything seems to be running smoothly). I'm a Matlab kind of guy. Hence the export to CSV, which I can read into Matlab.
I've spotted this problem when I ran a Matlab script to collect all wormholes. My idea was to simply find all systems that have no static connections. That finds all the J-systems and Thera, but to my surprise I found several extras like this one... |
Desmont McCallock
603
|
Posted - 2016.04.18 07:38:37 -
[42] - Quote
It's case 2.
From a look into the original SDE data it's missing data for the following constellations in that region also:
constellationID, constallationName 20000047L5-H1O 20000048J-4QD7 20000049LC-AQD 200000507PUP-K 20000051V8UW-M 20000052T7B-VB 20000053B-PHYN 200000549RW5-Z 20000055VL-OGL 20000056VW-JMB 20000057HN5-RL 20000058RO-AZT 20000059KCR-1E 20000060FL-TD4
So, file a bug report with CCP to have that fixed in the next release. |
Tristan Agion
Viziam Amarr Empire
134
|
Posted - 2016.04.18 09:30:46 -
[43] - Quote
Thanks for the quick response. I have now submitted a bug report! Actually, I found a lot more of these falsely isolated systems, I just didn't write this up:
'WF-1LM''D95-FQ''ZSPJ-K''U1F-86''T-P7A6''Y-T3JJ''F3R-IA''74-YTJ''8-RS3U''OVFN-N''1Q-BBM''WXNC-N''G-EA07''X-L6BO''D-PHUA''3-HXHQ''18A-NB''3-J5OQ''GYF-GD''W-6TS9''VIG-VR''KX-P5C''N-FJBK''2-4ZT5''NVN-6F''09-8TH''TI0-AX''7O-POM''L6Q-SX''BFJ-TB''ZZ7-L6''L-CHVW''X0LN-U''RQAE-M''7CO-SA''4G-E5A''A-VWK9''JQHP-4''6Q5K-5''P-MVFP''J-Z1UW''W477-P''NQ1-BL''K7A-G8''HP-PMX''6BN-K9''WLE-PY''EH-HXW''OS-RR3''V4-GZL''4C-Z91''RU-97T''1S-1V7''PE1-R1''Polaris''JB-007''USJ2-M''7M-RAL''LPBU-U''RF-342''J2V-XY''Z-JBTR''S-QNXH''S94-X8''J-YQEC''8MX-OR''97YC-C''V-AMD5''U-JC8X''1HH3-E''DUIU-Q''LQH0-H''FRW3-2''9MX-1C''IED-4U''N-9EOQ''6F3-TK''2E0P-2''U-ITH5''N-4G5L''RB-2EA''ZK5-42''YRZ-E4''A3-PAT''H55-2R''P6-DBM''9XI-0X''Q8T-MC''Z-YOJ9''4T4B-L''F-JB3H''XBO7-F''FI-449''UA7-U4''PZP1-D''R1KE-A''JGDF-B''1SR-HT''SQ-2XA''Z-FYJR''ZA6-9N''J1-6CJ''7H-Z5R''0RZ5-2''A9-NB6''LG1-TA''TNK-BQ''E2AX-5''HPE-KP''THS-MN''UBES-K''I-R8B0''QIW-TQ''WLL-QX''BJC4-8''PQA-9K''S5-U0R''CW-R71''QO-3LC''3E-ER7''REZ-YZ''OU-AIT''VYX2-I''5-P3CQ''M-FDTD''54-VNO''IAMZ-5''HD3-JK''PBXG-A''9-ERCP''KN7M-N''Z-D1DW''FO-3PJ''6-QXE6''N-FKXV''X7-8IG''R-G1SF''6-NCE7''WDJQ-G''JS3-RS''JX-T1W''CZ-CED''BKK4-H''Y-4V7U''L-TPN0''3-XORH''G1VU-H''W6H6-K''6-23NU''DVAR-P''J-JS0D''VR3-PS''LH-J8H''I9D-0D''HGB-C6''2L5-FI''RS08-B''4U-14I''H-EDXD''8-ULAA''KF1-DU''W-WQM5''G5J-LH''H7OL-I''TO21-U''RN-5K9''0M-M64''W5-SGC''8RV-1L''1C-TD6''YBYX-1''2-NF2Z''0Z-VHC''9-BUSQ''LQB-TC''II-1B3''6-HFD6''P3UD-M''LCN-0V''FX-XMW''G-N6MC''7-8XK0''90G-OA''DT-7EO''B-Y06L''HHQ-8L''Z-KPAR''8U-RZH''2RV-06''CLDT-L''QU7-EE''UC-X28''R79-I7''E-RPGP''ZV-KZO''NSE-U1''KER-EU''69A-54''M9-OS2''5V-YL6''8-UWFS''PQWA-L''BWO-UU''SQVI-U''T-YWDD''DLY-RG''T-C5A0''UP-L3Y''F-KBNV''JL-P9P''FR-RCH''FNS3-F''7BA-TK''IAWJ-X''50-TJY''3-CE1R''0IRK-R' |
Desmont McCallock
603
|
Posted - 2016.04.18 11:00:33 -
[44] - Quote
Yeah, you're posting the solar systems. I posted the constellations. |
Mr Mac
Dark Goliath
137
|
Posted - 2016.04.18 17:21:30 -
[45] - Quote
Hmm, I checked lastest SDE... I see the constellations and solarsystem there.
Edit. I can confirm missing data in mapSolarSystemJumps table |
Cryten Jones
C5 Flight The Serenity Initiative
124
|
Posted - 2016.04.21 13:20:47 -
[46] - Quote
Hi There,
Just starting to play with this.. I am trying to use the following connection string.. will it just not work or am I doing it wrong?
connectionString="Data Source=(LocalDB)\MSSQLLocalDB;AttachDbFilename="..\EvE Production.mdf";Initial Catalog=EveStaticData;Integrated Security=True;MultipleActiveResultSets=True;" providerName="System.Data.SqlClient"
The error I get is a hard crash with information saying that '.' can't be the start of the path
Unhandled Exception: System.Configuration.ConfigurationErrorsException: Configuration system failed to initialize ---> System.Configuration.ConfigurationErrorsException: Name cannot begin with the '.' character, hexadecimal value 0x2E. Line 8, position 103
|
Desmont McCallock
604
|
Posted - 2016.04.26 15:16:55 -
[47] - Quote
@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand. |
Tristan Agion
Viziam Amarr Empire
138
|
Posted - 2016.04.26 16:00:43 -
[48] - Quote
Desmont McCallock wrote:@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand. Does that follow? It still could be that the EVE client map is running on the same corrupted, compiled data that goes into the SDE, while in reality there are stargates in the system. My main comparison was against Dotlan, and Dotlan does show connections. Where does Dotlan get those from? I guess the objective test is to just fly to for example ZSPJ-K and see if I can find a stargate to D95-FQ as Dotlan says, or not, as the SDE says...
(Possibly some significance of this being "the CCP region" escapes me. I'm still pretty new to the game, after all. If this is a Singularity-like test region where things are more fluid, or if this is only accessible to CCP, do let me know.) |
Hel O'Ween
Men On A Mission
159
|
Posted - 2016.04.26 16:06:55 -
[49] - Quote
Hi Desmont,
using the new (now deemed "legacy") SDE linked by CCP Tellus, I got an error:
Unable to import ramTypeRequirements Reason was: Verletzung der PRIMARY KEY-Einschr+ñnkung 'ramTypeRequirements_PK'. Ein doppelter Schl++ssel kann in das dbo.ramTypeRequirements-Objekt nicht eingef++gt werden. Der doppelte Schl++sselwert ist (41590, 1, 38).
... which translates to
Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'. A duplicate key can't be inserted into the dbo.ramTypeRequirements object. The duplicate key value is (41590, 1, 38).
I've also crossposted over in the SDE changes thread: https://forums.eveonline.com/default.aspx?g=posts&m=6457455#post6457455
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
604
|
Posted - 2016.04.26 16:38:36 -
[50] - Quote
Hel O'Ween wrote:Hi Desmont, using the new (now deemed "legacy") SDE linked by CCP Tellus, I got an error: Unable to import ramTypeRequirements Reason was: Verletzung der PRIMARY KEY-Einschr+ñnkung 'ramTypeRequirements_PK'. Ein doppelter Schl++ssel kann in das dbo.ramTypeRequirements-Objekt nicht eingef++gt werden. Der doppelte Schl++sselwert ist (41590, 1, 38).
... which translates to Reason was: Violation of PRIMARY KEY constraint 'ramTypeRequirements_PK'. A duplicate key can't be inserted into the dbo.ramTypeRequirements object. The duplicate key value is (41590, 1, 38).
I've also crossposted over in the SDE changes thread: https://forums.eveonline.com/default.aspx?g=posts&m=6457455#post6457455 https://forums.eveonline.com/default.aspx?g=posts&m=6457513#post6457513 Workaround: - Open the blueprints.yaml file. - Navigate to line 169277 (blueprintTypeID: 41590) - Comment out lines 169277-78. - Run the tool again. |
|
Desmont McCallock
604
|
Posted - 2016.04.26 22:25:12 -
[51] - Quote
@Hel O'Ween After talking with CCP Tellus (https://forums.eveonline.com/default.aspx?g=posts&m=6458181#post6458181) I would suggest that you correct he blueprints.yaml file manually.
1. Delete lines 169277-78 2. Correct the quantity of typeID: 38 for manufacturing from 3 to 8. |
Hel O'Ween
Men On A Mission
159
|
Posted - 2016.04.27 16:15:21 -
[52] - Quote
Ah, didn't realize that the bluepint.yaml was culprit. Then the fix is easy for me: delete that YAML, as I don't use the BPO stuff at all.
As you explained longer ago, your tool is clever enough to skip non-existing YAMLs. All hail software that's smarter as its users (=me)!
EVEWalletAware - an offline wallet manager.
|
Darkblad
458
|
Posted - 2016.04.28 15:35:05 -
[53] - Quote
How curious
Did you also find that several tables of the legacy ms sql dump are mssing their content? dgmTypeAttributes (and its dgmAttributeTypes) is a bit annoying
NPE-ISD-Übersetzt!
|
Desmont McCallock
604
|
Posted - 2016.04.28 16:00:21 -
[54] - Quote
Darkblad wrote:How curiousDid you also find that several tables of the legacy ms sql dump are mssing their content? dgmTypeAttributes (and its dgmAttributeTypes) is a bit annoying.I just got the info from Fuzzsteve that there was another more recent SDE, so I got that one Not true with Citadel 1.1 SDE.
|
salacious necrosis
Federal Defense Union Gallente Federation
16
|
Posted - 2016.04.29 11:59:59 -
[55] - Quote
Hey Des,
Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?
Thanks! And thanks again for the great tool!
|
Desmont McCallock
604
|
Posted - 2016.04.29 12:02:06 -
[56] - Quote
salacious necrosis wrote:Hey Des,
Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?
Thanks! And thanks again for the great tool!
I'm already working on a version that can import the new and the legacy format but it will take some time.
|
salacious necrosis
Federal Defense Union Gallente Federation
16
|
Posted - 2016.04.29 12:03:34 -
[57] - Quote
Desmont McCallock wrote:salacious necrosis wrote:Hey Des,
Do you need to cut a new release of your tool with fixes for the new SDE? Or are you planning to leave it as-is for now?
Thanks! And thanks again for the great tool!
I'm already working on a version that can import the new and the legacy format but it will take some time.
Awesome. Happy to help beta test when ready. |
Golden Gnu
EVE University Ivy League
181
|
Posted - 2016.04.30 11:05:39 -
[58] - Quote
@Desmont McCallock Thank you for the workaround for blueprints.yaml And as always; thank you for this great program.
Creator of jEveAssets - the asset manager
"Download is the meaning of life, upload is the meaning of intelligent life"
|
Desmont McCallock
606
|
Posted - 2016.04.30 12:24:48 -
[59] - Quote
Tristan Agion wrote:Desmont McCallock wrote:@Tristan Agion While I was dealing with the new SDE format I realized that region UUA-FA is the CCP region. If you look at the EVE client map you will see that there are only a few solar systems that are connected (i.e. have stargates). Therefor, the data are correct and you should close the bug report cause your claim doesn't stand. Does that follow? It still could be that the EVE client map is running on the same corrupted, compiled data that goes into the SDE, while in reality there are stargates in the system. My main comparison was against Dotlan, and Dotlan does show connections. Where does Dotlan get those from? I guess the objective test is to just fly to for example ZSPJ-K and see if I can find a stargate to D95-FQ as Dotlan says, or not, as the SDE says... (Possibly some significance of this being "the CCP region" escapes me. I'm still pretty new to the game, after all. If this is a Singularity-like test region where things are more fluid, or if this is only accessible to CCP, do let me know.) Edit: OK, on looking around on the web and Dotlan some more, it seems that my "objective" test will fail simply because one cannot jump into this region directly in the first place. Does anybody know whether any wormhole connections ever form to UUA-FA? Still wondering how Dotlan got its data... Sorry for the late reply. My eye just now caught your post. UUA-FA has always been the CCP region meaning that in that region CCP tests things in production. No capsuleer can access that region unless (s)he has been transferred there by CCP. Dotlan hasn't updated its Universe data, since who knows when, probably from the time Wormholes where introduced, simply because there is no reason to do so as these data are static and don't change frequently.
Now if your O.C.D. (joke) impulses you for Dotlan and SDE to match, contact Wollari and ask him to update his DB. |
Desmont McCallock
609
|
Posted - 2016.05.05 16:25:45 -
[60] - Quote
I'm sadden to announce that EVESDEToSQL will NOT support the new SDE format until CCP responds to my post or fixes at least the data inconsistencies between the legacy and the new format (also mentioned in my post).
@CCP devs Not answering is not a nice way to treat customers not to mention long-time customers. |
|
Desmont McCallock
635
|
Posted - 2016.05.09 14:36:58 -
[61] - Quote
Releasing version 3.5.0-RC1.
- Added support to import the new SDE format. - Added ability to handle duplicate quantity entries in blueprints.yaml file.
Notice: This version will present you some WARNINGS due to the following facts:
1. The names for the mapDenormalize table are imported via a lookup on the invNames table or file. 2. The names and descriptions for the mapLandmarks table are imported via a lookup on the trnTranslation table or file.
You will notice some data inconsistencies in the mapDenormalize table for the names in comparison with the legacy format. Don't blame the software. |
Desmont McCallock
635
|
Posted - 2016.05.14 18:56:34 -
[62] - Quote
Releasing version 3.5.0.
This the last release from me, folks.
Take care in RL.
Fly safe o7 |
Lost Hamster
Hamster Holding Corp
156
|
Posted - 2016.05.15 12:05:37 -
[63] - Quote
Sir, you are a star!
Thanks a lot! |
|
|
|
Pages: 1 2 3 :: [one page] |