Pages: [1] 2 3 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Desmont McCallock
569
|
Posted - 2015.11.15 13:09:42 -
[1] - Quote
Introducing the next generation of EVE SDE handling tool (formely known as EVEMonSDEExternalsToSql).
I'm presenting to you EVESDEToSQL v3. Version 3 is the evolution of the previous version, containing all the goodies of v2, plus the ability to export the EVE SDE to the most popular SQL formats and more.
The tool source code has been separated from EVEMon and now resides in its own repository.
The compiled version can be found for downloading here.
The latest version is: 3.0.0
Direct dl link: EVESDEToSQL-3.0.0.zip
Notice: The SQL data dump, yaml and SQLite db files are not distributed along with the tool. You will have to get them from the community toolkit page.
--Features--
-Importation
* Restores the data dump, yaml and sqlite db files back into an SQL Server (no need to restore the data dump via SSMS). * Imports any SDE (literally any, but in case you find an SDE with issues please let me known) * Supports command line arguments (use 'help' to see the list) * Cleaner config file (it now only contains the needed connection info to your SQL Server, and it's the only thing that you may need to modify) * Ulta-High Speed (it now takes less than 5 minutes to import the entire SDE, depending on your machine's cpu power) * One place for the SDE file(s) (just drop the SDE zip file as is or the files contained in the SDE zip file, into the 'SDEFiles' folder)
-Exportation
* Exports an SQL Server DATADUMP backup file * Exports an SQLite database file * Exports an MS Access database file ('mdb' as default, use '/ace' switch for 'accdb') * Exports an MySQL dump file * Exports an PostgreSQL dump file * Exports individual CSV (semicolon seperated version) files * Ability to export specific tables in any dump format
Supported OS: Windows XP, Vista, 7, 8, 8.1, 10
Exportation Usage Examples
SQL Server dump: EVESDEToSQL export -sql [/et: listOfTables] MySQL dump: EVESDEToSQL export -mysql [/et: listOfTables] PostgreSQL dump: EVESDEToSQL export -postgresql [/et: listOfTables] SQLite dump: EVESDEToSQL export -sqlite [/et: listOfTables] Ms Access dump: EVESDEToSQL export -access [/ace] [/et: listOfTables] CSV dump: EVESDEToSQL export -csv [/et: listOfTables]
Requirements
* .NET 4.0 or greater * SQL Server instance (any version, even Express)
Yaml files mapping:
blueprints.yaml -> invBlueprintTypes, ramTypeRequirements categoryIDs.yaml -> invCategories, translationTables, trnTranslationColumns, trnTranslations certificates.yaml -> crtClasses, crtCertificates, crtRecommendations, crtRelationships graphicIDs.yaml -> eveGraphics groupIDs.yaml -> invGroups, translationTables, trnTranslationColumns, trnTranslations iconIDs.yaml -> eveIcons skinLicenses.yaml -> sknLicenses skinMaterials.yaml -> sknMaterials skins.yaml -> sknSkins tournamentRuleSets.yaml -> ignored typeIDs.yaml -> invTypes, dgmMasteries, dgmTypeMasteries, dgmTraits, dgmTypeTraits, translationTables, trnTranslationColumns, trnTranslations
Sqlite files mapping:
universeDataDx.db -> mapCelestialStatistics, mapConstellationJumps, mapConstellations, mapDenormalize, mapJumps, mapLandmarks, mapLocationScenes, mapLocationWormholeClasses, mapRegionJumps, mapRegions, mapSolarSystemJumps, mapSolarSystems
Known Issues:
None (so far)
Enjoy responsibly. |
Desmont McCallock
569
|
Posted - 2015.11.20 09:27:45 -
[2] - Quote
FAQ
Q:I don't want the name of the database to be 'EVEStaticData'.
A: Edit the EVESDEToSQL.exe.config file and change the value of connection's string 'Initial Catalog' to name the database differently.
Q:I'm getting the error message Connection to SQL Server 'EveStaticData' database: Failed Reason was: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
A: Edit the EVESDEToSQL.exe.config file and adjust the value of connection's string 'Data Sourse' to point to your SQL Server instance (the tool assumes that the SQL Server instance is named SQLEXPRESS).
Q:I'm getting the error message Connection to SQL Server 'EveStaticData' database: Failed Reason was: Cannot open database "EveStaticData" requested by the login. The login failed. Login failed for user 'computerName\user'.
A: This message appears when you try to export to a dump format. Do an import first (EVESDEToSQL import). |
Desmont McCallock
569
|
Posted - 2015.11.20 10:56:45 -
[3] - Quote
Releasing version 3.0.1.
- Includes several bug fixes and improved error handling. |
Golden Gnu
EVE University Ivy League
174
|
Posted - 2015.11.23 14:21:23 -
[4] - Quote
Thanks Desmont! :) I'm still only using the import feature, but, it's amazing how this software always works without any problem. You must have a talent for writing high quality code o7
Download is the meaning of life, upload is the meaning of intelligent life
http://eve.nikr.net - home of jEveAssets
|
Desmont McCallock
572
|
Posted - 2015.11.23 16:48:34 -
[5] - Quote
Golden Gnu wrote:Thanks Desmont! :) I'm still only using the import feature, but, it's amazing how this software always works without any problem. You must have a talent for writing high quality code o7
Thank you for the compliment.
|
GrayLensman
Galactic Protectorate
0
|
Posted - 2015.12.01 01:52:41 -
[6] - Quote
Desmont,
I have attempted to run your utility several times but am clearly missing something. I have attached two images:
(1) showing the location of EVESDEToSQL and the contents of the SDEFiles folder first...
Picture 1
(2) Then showing the results and that EVESDEToSQL is not able to locate the SDE files...
Picture 2
What am I missing? Thanks.
Gray Lensman |
Desmont McCallock
574
|
Posted - 2015.12.01 08:10:51 -
[7] - Quote
It looks like the unzipping is failing. I'll have a look at it. Meanwhile don't use the zip file inside the folder but just unzip the files in it and remove the zip file afterwards. |
Desmont McCallock
574
|
Posted - 2015.12.01 08:37:31 -
[8] - Quote
Released version 3.1.0.
Change log * Added support for using SQL Server Authentication. * Fixed a bug when trying to unzip the SDE zip file. |
GrayLensman
Galactic Protectorate
0
|
Posted - 2015.12.01 16:04:51 -
[9] - Quote
3.1 tested and confirmed, results of import below provided in 57.38 seconds... this really is excellent work, nice job...
Considering the amount of labor you saved me I have contributed 50m isk to the Desmont McCallock Software Services Fund... I encourage other pilots to do the same.
D:\Dev\Data\EVESDEToSQL-3.1.0>evesdetosql import Unzipping SDE zip file: Successful
Importing SQL data dump to 'EveStaticData' database... 100 % in 0:00:02.2645004
Connection to SQL Server 'EveStaticData' database: Successful
Importing categoryIDs.yaml... 100 % in 0:00:00.1473313 Importing groupIDs.yaml... 100 % in 0:00:00.2944629 Importing graphicIDs.yaml... 100 % in 0:00:00.3124175 Importing iconIDs.yaml... 100 % in 0:00:00.1020952 Importing skins.yaml... 100 % in 0:00:00.0881721 Importing skinMaterials.yaml... 100 % in 0:00:00.0217246 Importing skinLicenses.yaml... 100 % in 0:00:00.0708117 Importing typeIDs.yaml... 100 % in 0:00:16.3830429 Importing certificates.yaml... 100 % in 0:00:00.2589833 Importing blueprints.yaml... 100 % in 0:00:02.8515145
Connection to SQLite 'main' database: Successful
Importing mapCelestialStatistics... 100 % in 0:00:09.2347243 Importing mapConstellationJumps... 100 % in 0:00:00.0475246 Importing mapConstellations... 100 % in 0:00:00.0408733 Importing mapDenormalize... 100 % in 0:00:19.1871954 Importing mapJumps... 100 % in 0:00:00.0954077 Importing mapLandmarks... 100 % in 0:00:00.0177922 Importing mapLocationScenes... 100 % in 0:00:00.0150901 Importing mapLocationWormholeClasses... 100 % in 0:00:00.0198079 Importing mapRegionJumps... 100 % in 0:00:00.0171859 Importing mapRegions... 100 % in 0:00:00.0250435 Importing mapSolarSystemJumps... 100 % in 0:00:00.22791 Importing mapSolarSystems... 100 % in 0:00:00.2612589
Disconnection from SQLite 'main' database: Successful
Reorganizing indexes of database 'EveStaticData'... 100 % in 0:00:05.085969
Shrinking database 'EveStaticData'... in 0:00:00.1521892
Disconnection from SQL Server 'EveStaticData' database: Successful
Importing files completed in 0:00:57.3862157
Press any key to exit. D:\Dev\Data\EVESDEToSQL-3.1.0> |
Desmont McCallock
574
|
Posted - 2015.12.01 19:23:50 -
[10] - Quote
@GrayLensman Very much appreciated. |
|
Desmont McCallock
585
|
Posted - 2015.12.07 10:23:25 -
[11] - Quote
For those having issue with using the SDE zip file, make sure that the file is named according to CCP's naming convention 'ReleaseName_VersionMajor.VersionMinor_VersionBuild_db.zip' (which they don't seem to honor in some cases). |
Hel O'Ween
Men On A Mission
153
|
Posted - 2015.12.10 09:51:28 -
[12] - Quote
Continuing from here
Running the tool with ...
EVESDEToSQL import -norestore -noyaml
... still fails with the already mentioned timeout and produces the below content in trace.txt:
Quote: EVESDEToSQL: Started Native library pre-loader is trying to load native SQLite library "C:\DATEN\x86\SQLite.Interop.dll"... Deleting file C:\DATEN\SDEFiles\universeDataDx.db System.IO.IOException: Der Prozess kann nicht auf die Datei "C:\DATEN\SDEFiles\universeDataDx.db" zugreifen, da sie von einem anderen Prozess verwendet wird. bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) bei System.IO.File.InternalDelete(String path, Boolean checkHost) bei System.IO.File.Delete(String path) bei EVESDEToSQL.Utils.Util.DeleteSDEFilesIfZipExists() in EVESDEToSQL\Utils\Util.cs:Zeile 184.
Der Prozess kann nicht auf die Datei "C:\DATEN\SDEFiles\universeDataDx.db" zugreifen, da sie von einem anderen Prozess verwendet wird.
translates to
The process can't access file "C:\DATEN\SDEFiles\universeDataDx.db", becuase it is in use by another process.
I peeked at Util.cs and it seems it tries to delete the SQLite DB while it's still in use somewhere. My C# is too weak to determine the real cause, but I suspect this is a follow-up error of the import issue, which leaves the SQLite DB still connected/open, and this deletion exception happens during the "normal" cleanup operation when closing the application.
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
586
|
Posted - 2015.12.10 13:42:59 -
[13] - Quote
Hel O'Ween wrote:I peeked at Util.cs and it seems it tries to delete the SQLite DB while it's still in use somewhere. My C# is too weak to determine the real cause, but I suspect this is a follow-up error of the import issue, which leaves the SQLite DB still connected/open, and this deletion exception happens during the "normal" cleanup operation when closing the application. This is a normal error as the tool tries to delete the files it had unzipped once you close it. The fact that it's still failing to import only the sqilte DB is troublesome. Just for the record, I can't reproduce it on my machine but I'll investigate this behavior further.
|
Hel O'Ween
Men On A Mission
153
|
Posted - 2015.12.10 18:06:20 -
[14] - Quote
Desmont McCallock wrote: The fact that it's still failing to import only the sqilte DB is troublesome. Just for the record, I can't reproduce it on my machine but I'll investigate this behavior further.
It does import the sqlite quite find, it just chokes on that single mapDenormalize table after the counter goes up to 100% for it.
I just dropped mapDenormalize from the sqlite and tried the import again: worked like a charm. Yeah, mapDenomralize is the biggest table, but it's still "peanuts" for any really RDBMS.
Thanks, but the good ol' EVEMonSDEExternalsToSql.exe already did the job for me.
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
589
|
Posted - 2015.12.19 10:23:34 -
[15] - Quote
Iterating again on your problem here is what I found so far:
Fact: Memory consumption is maximum 500MB during the entire operation. Conclusion: Therefore it's mot a memory consumption issue. Action: No action will be taken.
Fact: The bulk insert timeout is fixed at 30 seconds and is not linked to the connection timeout setting in the connection string. Conclusion: Most likely the problem lies there, as from looking at the times it takes on your machine to insert the other tables, MapDenormalize exceeds the timeout and any offset it might have. Action: I will add some code to link the connection timeout set in the connection string to the bulk insert timeout. |
Desmont McCallock
589
|
Posted - 2015.12.19 10:32:15 -
[16] - Quote
Released version 3.1.1.
Change Log * Improved search algorithm for the SDE zip file. * Linked the bulk insert operation timeout to the connection timeout if it's set and exceeds the default value. |
Desmont McCallock
589
|
Posted - 2015.12.19 10:39:26 -
[17] - Quote
@Hel O'Ween Try again setting 'Connection Timeout=60' in the connection string. |
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.19 10:51:00 -
[18] - Quote
I'm not having access to my dev machine right now, will try that on monday.
Regardless of the outcome - many thanks for your ongoing efforts. Much appreciated, Desmont!
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
590
|
Posted - 2015.12.19 11:20:30 -
[19] - Quote
Hel O'Ween wrote:I'm not having access to my dev machine right now, will try that on monday.
Regardless of the outcome - many thanks for your ongoing efforts. Much appreciated, Desmont! Sure thing. Thanks for your kind words. |
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.21 12:12:04 -
[20] - Quote
And surely enough, that fixed it.
Thanks, Desmont.
EVEWalletAware - an offline wallet manager.
|
|
Desmont McCallock
591
|
Posted - 2015.12.21 12:29:49 -
[21] - Quote
Hel O'Ween wrote:And surely enough, that fixed it. Thanks, Desmont. Thanks for confirming.
|
salacious necrosis
Federal Defense Union Gallente Federation
14
|
Posted - 2015.12.24 18:14:27 -
[22] - Quote
Hi Des,
Used your exporter for the first time today. Everything worked well except for SQLite import. To get that to work I had to copy x86/SQLite.Interop.dll to the main EVESDEToSQL directory. Otherwise, the importer fails with a trace.txt like this:
Quote: EVESDEToSQL: Started Native library pre-loader is trying to load native SQLite library "C:\Users\mark_000\projects\sdeimport\EVESDEToSQL\x86\SQLite.Interop.dll"... Native library pre-loader is trying to load native SQLite library "C:\Users\mark_000\projects\sdeimport\EVESDEToSQL\x86\SQLite.Interop.dll"... System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.DllNotFoundException: Unable to load DLL 'SQLite.Interop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_none(SQLiteConfigOpsEnum op) at System.Data.SQLite.SQLite3.StaticIsInitialized() at System.Data.SQLite.SQLiteLog.Initialize() at System.Data.SQLite.SQLiteConnection..ctor(String connectionString, Boolean parseViaFramework) at System.Data.SQLite.SQLiteConnection..ctor(String connectionString) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at EVESDEToSQL.Providers.DbConnectionProvider.CreateConnection(Type connectionType, String nameOrConnectionString) in EVESDEToSQL\Providers\DbConnectionProvider.cs:line 97 at EVESDEToSQL.Providers.SqliteConnectionProvider..ctor(String nameOrConnectionString) in EVESDEToSQL\Providers\SqliteConnectionProvider.cs:line 22 at EVESDEToSQL.Importer.ImportSqlite() in EVESDEToSQL\Importer.cs:line 185 at EVESDEToSQL.Importer.ImportSDEFiles(String[] args) in EVESDEToSQL\Importer.cs:line 135 at EVESDEToSQL.Importer.Start(String[] args) in EVESDEToSQL\Importer.cs:line 90
System.DllNotFoundException: Unable to load DLL 'SQLite.Interop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at System.Data.SQLite.UnsafeNativeMethods.sqlite3_config_none(SQLiteConfigOpsEnum op) at System.Data.SQLite.SQLite3.StaticIsInitialized() at System.Data.SQLite.SQLiteLog.Initialize() at System.Data.SQLite.SQLiteConnection..ctor(String connectionString, Boolean parseViaFramework) at System.Data.SQLite.SQLiteConnection..ctor(String connectionString)
My box is x64 but it didn't try to load from that directory at all. Anyway, happy to help debug further if you provide instructions.
|
Desmont McCallock
591
|
Posted - 2015.12.25 13:13:10 -
[23] - Quote
Although I suspect it's mostly a SQLite issue I'll surely look into it. |
Desmont McCallock
591
|
Posted - 2015.12.25 13:14:22 -
[24] - Quote
Is there any interest in providing an SQL CE (SQL Server Compact 4) dump? |
salacious necrosis
Federal Defense Union Gallente Federation
14
|
Posted - 2015.12.25 18:34:07 -
[25] - Quote
Desmont McCallock wrote:Although I suspect it's mostly an SQLite issue I'll surely look into it.
It doesn't matter if your box is x64. The app is set to run as x86 (to provide support fro MS Access mdb files).
Never the less, I would much appreciate if you could tell me what OS your box is running on.
Running Windows 8.1 Pro, with .NET 4.0.30319
I don't have any of the SQLite dll's in my path except for the those included in EVESDEToSQL. I DO have the sqlite3 CLI binary in my path, but I don't think that should affect anything.
|
Desmont McCallock
591
|
Posted - 2015.12.26 14:08:12 -
[26] - Quote
So I checked the tool again and I can't find anything wrong with it.
The only way I could reproduce the exact exception you are reporting, is to rename the 'x64' folder to 'x86', to fool the app so it tries to load the 64bit version of the interop library.
So for some reason on your machine the tool although it says it's looking into 'x86' it tries to load the 64bit dll or for some unknown reason to me the interop dlls got mixed up.
I suggest you delete the folder where the tool currently resides and re-download it.
Just for the reference here is the Dynamic-Link Library Search Order that Windows uses.
- The directory from which the application loaded.
- The system directory. Use the GetSystemDirectory function to get the path of this directory.
- The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched.
- The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
- The current directory.
- The directories that are listed in the PATH environment variable. Note that this does not include the per-application path specified by the App Paths registry key.
|
Hel O'Ween
Men On A Mission
154
|
Posted - 2015.12.28 11:10:30 -
[27] - Quote
I'm very new to this .NET stuff, so I might be totally of here, but I remember something about assemblies being registered in the GAC ... might something like this cause the lookup order to not working as expected?
EVEWalletAware - an offline wallet manager.
|
Desmont McCallock
591
|
Posted - 2015.12.28 12:41:42 -
[28] - Quote
Hel O'Ween wrote:I'm very new to this .NET stuff, so I might be totally of here, but I remember something about assemblies being registered in the GAC ... might something like this cause the lookup order to not working as expected? You are correct Hel O'Ween. It's a possibility.
|
Desmont McCallock
598
|
Posted - 2016.02.05 11:46:08 -
[29] - Quote
Released version 3.2.0.
ChangeLog Updated SQLite nuget package. Updated YamlDotNet nuget package. Included missing SMO assembly (was though available in GAC if SQL Server was installed on the machine). |
Grenval Truvaki
Machiavellian Empire Fidelas Constans
0
|
Posted - 2016.03.01 18:14:26 -
[30] - Quote
Hi Desmont,
This tool seems really useful to me! However i've run into a bit of an issue.
Reason was: System.Data.SqlClient.SqlError: Cannot open backup device 'C:\Users\ Simon\Downloads\EVESDEToSQL-3.2.0\SDEFiles\DATADUMP201602091443.bak'. Operating system error 3(The system cannot find the path specified.).
Press any key to exit.
Any idea's to what I've done wrong? haha |
|
|
|
|
Pages: [1] 2 3 :: one page |
First page | Previous page | Next page | Last page |