Pages: [1] 2 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.01 09:53:00 -
[1] - Quote
EveOrg App has been updated (finally) to use the new API key system. Other changes include...
* no registration required, fully free to use personally, corp or alliance wide * improved API data processing, now more efficient and reliable * API data scraping expanded to encompass all available API data (except server status)
The EOApp runs under .Net (v3.5) on Windows (AFAIK it does not work on mono). It does three things - in a fancy multi-threaded way ...
API 2DB You provide a SQL database and put some API keys in it. API2DB will continually scrape the API for all available data for all the keys and keep the SQL database up to date with the latest data 24/7. Some data (like market orders and sovereignty) also build historically to provide a useful data mine.
Every piece of data available from the API is scraped. Assets, Contracts, Market Orders, Notifications, Research + Industry Jobs, Contacts, Standings, Main, lock stock the f@#!ing lot. All public data is scraped, sovereignty, map kills/jumps, alliance list, etc. Even useful anonymous data is scraped about characters and corps - e.g. your wallet transaction data reports selling an item to character "EveOrg" then as much data about EveOrg as possible is scraped from public API calls (like CharacterInfo).
What can you use this for? What can't you use it for! Monitor and plan your trading. Gather intel on corp applicants. Alliance wide detection of hostile activities. Backend to a website. Monitor corp contracts. Easily queryable security logs. Anything you can dream up to do with the data is just a SQL query away :)
Market 2DB Monitors the game client cache folder for market data. As soon as a new market cache file appears the data is added to a table in the SQL database. You can rapidly build up a database of accurate up to date market orders/prices/etc by clicking through the market in-game.
API Proxy Acts as a proxy for the API. Point your other API consuming applications at it instead of CCP and always get the latest data (rather than the API rejecting apparently repeated calls).
It's pretty much at late beta stage. Thanks to some early beta testers from a thread in the old forums it now fairly reliable and efficient So I'm making it available for public consumption. Free of charge. No registration. Donations of isk are of course appreciated
Please report bugs in this thread, or to me via Eve mail, of drop into the in-game channel eveorg.net. Hope people find it useful! |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.01 09:56:00 -
[2] - Quote
I would also be interested in hearing from people wanting to integrate other tools/services. Like centralized market data etc. |
Jenn Makanen
Science and Trade Institute Caldari State
36
|
Posted - 2011.11.01 13:03:00 -
[3] - Quote
Have you thought about using odbc for the database connection? (Disclaimer: I have no idea how hard this would be)
I've been playing with the python scrapers to get stuff into a remote mysql DB, and this tool looks like it would handle it better. |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.01 22:05:00 -
[4] - Quote
The connection is over ODBC, you give it an ODBC connection string. It talks SQL to the DB - it's expecting an MS SQL DB not sure how it would fair with other DBs. Please try and let me know :)
It works fine with the free version of MS SQL Express if it's a cost thing. |
Jenn Makanen
Science and Trade Institute Caldari State
36
|
Posted - 2011.11.01 22:47:00 -
[5] - Quote
Oh neat I'll give it a shot later.
Not a cost thing in this case. Just an accessibility thing. If I load the data onto the mysql database, I can get at it from work, as it's on a dedicated server. Gives me more time to play with it, that's not Eve time |
Jenn Makanen
Science and Trade Institute Caldari State
36
|
Posted - 2011.11.01 23:29:00 -
[6] - Quote
damn. can't get a connection string it'll like.
Oh well. Time to install sql server, and see about a nightly upload from that |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.02 00:46:00 -
[7] - Quote
None of this any help? MySQL connection strings
Or shoot me an evemail with details of the server if it's publically accessible I'll see what I can do.
I'll spend some time this weekend setting up mySQL on a linux VM and see if I can make it work(tm). |
Jenn Makanen
Science and Trade Institute Caldari State
36
|
Posted - 2011.11.02 00:58:00 -
[8] - Quote
If I remember right I've got a full sql server license kicking around somewhere. (microsoft academy has it's uses ) so I'll give that and some replication software a go. I did give the various connection string types a go, but either they died on the driver or provider portion, or just tried using sql server.
Nothing important really. Just me being a little lazy and not working on the python sample I've got. If you do get it working, I will be grateful however |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.02 01:06:00 -
[9] - Quote
No need for a full license, SQL Express is up to the task and completely free.
I will have a go at making it work with MySQL, if nothing else the more options people have for deployment the more useful it is - and the more chance of generous isk donations being heaped upon me
Plus I've been meaning to try this linux thing for a while... |
EveOrg
EveOrg.Net
0
|
Posted - 2011.11.05 02:36:00 -
[10] - Quote
Jenn Makanen wrote:Have you thought about using odbc for the database connection? (Disclaimer: I have no idea how hard this would be)
I've been playing with the python scrapers to get stuff into a remote mysql DB, and this tool looks like it would handle it better.
New version of EveOrg App out - supporting MySQL :) Now you can configure for MSSQL or MySQL compatibility. All your API data in the database of your choice!
Also fixed a few bugs and that
EveOrg App
|
|
6ie
State War Academy Caldari State
0
|
Posted - 2012.01.15 02:18:00 -
[11] - Quote
Looks awesome. Setting this up to talk to my MYSQL DB right now. Will let you know how it works out. |
Kepuh Harjo
whips chains and ballgags Care Factor
0
|
Posted - 2012.01.18 10:36:00 -
[12] - Quote
Hi EveOrg!
Great app, really love it!
I sent you several in-game mails with bugs/fixes/etc... but havn't received anything back yet. I've also joined the in-game support channel, but likely having timezone issues with connecting.
I'd love to work with you on the project (have done my own .net MSSQL project and was ready to start again when I found yours).
As a note, dumper.exe crashes every single time it tries to read a file. Does this work for anyone? (Win 7 64) |
6ie
State War Academy Caldari State
0
|
Posted - 2012.01.19 13:58:00 -
[13] - Quote
Market 2DB doesn't seem to work:
80c.cache... ERROR Unknown column 'cacheTS' in 'field list' 3c90.cache... 0 orders (PS.Q=0) 2448.cache... 0 orders (PS.Q=0) 81f7.cache... ERROR Unknown column 'cacheTS' in 'field list' f1a8.cache... ERROR Unknown column 'cacheTS' in 'field list' c900.cache... 0 orders (PS.Q=0) a9e3.cache... 0 orders (PS.Q=0) a87f.cache... 0 orders (PS.Q=0)
|
Kepuh Harjo
whips chains and ballgags Care Factor
0
|
Posted - 2012.01.19 16:48:00 -
[14] - Quote
so umm, i'm a little dumb at times :)
http://dl.stackfoundry.org/libevecache/
when i went here to download, i took the top file, without noticing the date... guess I just assumed it was sorted by date desc...
silly me :)
[16:39:05] 6ietrader > this is the dumper i use http://dl.stackfoundry.org/libevecache/libevecache-win32-bin-0.2.1.zip
6ie to the rescue via the in-game chat channel and now my public orders are being populated!
VICTORY IS MINE!!! (or well, his... but whatever) :) |
Aineko Macx
Royal Amarr Institute Amarr Empire
136
|
Posted - 2012.02.06 20:37:00 -
[15] - Quote
Bug when running the MySQL DB under a *nix system: In these systems, table names are case sensitive, but EONapp creates all lower case tables, then calls tables with camel case name, and consequently errors out.
Initial startup checks... Validating API database... Initialising schema... OK. ... EveOrg DB appears OK. Validating Eve database... OK. ... Eve DB appears OK. Validating Market database... OK. Initialising... OK. Validating static data with EveOrg.Net... AgentIDs ERROR! Table 'eonapp.eoncache_AgentIDs' doesn't exist |
Kepuh Harjo
whips chains and ballgags Care Factor
0
|
Posted - 2012.02.07 21:57:00 -
[16] - Quote
i have the same issues... just with a binary collation MS SQL Server
first i had to rename all the EONCACHE tables to lower case, just to get the test config to pass, but now when I try to add api keys it complains about the ccpuser table not found.
the table name in the db is ccpUser yet the code is looking for ccpuser
I was able to make the nessary changes to the last version to handle my case-sensitive server collation, but this version seems to have alot more mis-matched upper/lower case issues :(
can't use this version at all sadly :( |
6ie
The Kiwis
3
|
Posted - 2012.02.07 23:01:00 -
[17] - Quote
An issue with the new version when enabling Market2DB:
Exception: FUNCTION eonappdb.dateadd does not exist |
Imuran
Zentor Industries
10
|
Posted - 2012.02.09 16:39:00 -
[18] - Quote
Falling at the first hurdle here
Validating API database... Initialising schema... ERROR! Can't create table 'EonAppDB.xmlcachetimes' (errno: 150)
MySQL database running on a QNAP NAS
|
Nathan Gorden
Strategic Investment Group
0
|
Posted - 2012.02.18 06:04:00 -
[19] - Quote
6ie wrote:An issue with the new version when enabling Market2DB:
Exception: FUNCTION eonappdb.dateadd does not exist
I use MySQL and I have this exact same problem. I sent a message to the developer, but I have not gotten a response back. I really hope this gets fixed soon as this is an extremely valuable tool.
|
Nathan Gorden
Strategic Investment Group
0
|
Posted - 2012.02.27 05:45:00 -
[20] - Quote
An update to this Application. IT DOES WORK!
I spent hours trying to get the MySQL feature to work...I eventually gave up. The MySQL portion of this App is currently broken. I took the time to install MS SQL Server 2008 Express and the Market 2DB function works just fine. Likewise with Market Sync, it works. As much as I would like to use MySQL, MSSQL will have to suffice. |
|
EveOrg
EveOrg.Net
1
|
Posted - 2012.03.18 11:02:00 -
[21] - Quote
Hi, sorry been a while since last update. Had to go away for a month abroad to work. Back and with an update to EveOrg. Will attempt to answer some questions posted above...
dumper.exe crashing. I found a new version of dumper from the libevecache website fixed that (something changed!)
Kepuh Harjo, having to recapitalise all the fieldnames in code would be a big job, maybe next update sorry.
6ie, fixed 'cacheTS' bug in this version.
6ie, Nathan Gorden, fixed dateadd bug. Sorry you had to go to the trouble of installing ms-sql ;)
Sorry about the lowecase/camelcase problems. I will address them in a future update. |
6ie
The Kiwis
3
|
Posted - 2012.03.18 11:48:00 -
[22] - Quote
Cool - will install soon and report back! |
Xelia Otsada
Omnicorp Unlimited
0
|
Posted - 2012.04.14 17:40:00 -
[23] - Quote
Good morning. I was looking to try your application, EONAPP v0.9.8.4 . I was able to create the MS SQL and users without a problem.
However, when I go to test the config I get the following error after it passes the DB checks:
Starting version 0.9.8.4 at 2012-04-14 17:24:45 (UTC) Configuration EODB: Data Source=xxxx\xxxxxxx; Initial Catalog=EONAppDB;user id=xxx;password=; is: MSSQL EveDD: Data Source=xxxx\xxxxxxx; ;Initial Catalog=Crucible_1.6_65449;user id=xxx;password=; is: MSSQL MarketDB: Data Source=xxxx\xxxxxxx;Initial Catalog=EONAppDB;user id=xxx;password=; is: MSSQL Data: {directory path removed}EONAppDataStore Admin: xxxx SMTP: xxxx API: http://api.eve-online.com ProxP: 30303 Stats: True Dumper: {directory path removed}dumper.exe Eve: {directory path removed}c_games_xxxx_tranquility MSyncID: not configured! Initial startup checks... Validating API database... OK. ... EveOrg DB appears OK. Validating Eve database... OK. ... Eve DB appears OK. Validating Market database... OK. Initialising... OK. Validating static data with EveOrg.Net... AgentIDs downloading new data (remote)... ERROR! The remote server returned an error: (404) Not Found.
I tried disabling my firewall and still encountered the same error. Can you provide details on what remote server it is trying to go to get the AgentIDs download?
Cheers, Xelia |
6ie
The Kiwis
4
|
Posted - 2012.04.27 00:07:00 -
[24] - Quote
hmmm i'm getting this as of today..
Starting version 0.9.8.4 at 2012-04-27 00:05:29 (UTC) Configuration EODB: server=c... ... ... Initialising... OK.Validating static data with EveOrg.Net... AgentIDs checking... ERROR!Conversion from string "\StaticData" to type 'Integer' is not valid.
Due to the Escalation DB changes? |
EveOrg
EveOrg.Net
1
|
Posted - 2012.05.07 06:21:00 -
[25] - Quote
Yes, more than likely the DB changes. Giant new update has been released that takes care of that, and some other things. Including..
Totally Refactored Market Sync
Massively more scalable and efficient market sync feature. Sync your market price database with the central server and get universe wide price data (assuming someone has uploaded it). Yes, a bit like Eve-Central et al.
This central price data is also available to anyone who wants to query it with some code/script, instructions on the Market Prices page. Or you can just let the app do the work then query your own DB yourself.
I just noticed a forum thread about combining market data from the other market data websites, guess I'll have a read of that and see what's going on there... |
Khorkrak
War Tribe
2
|
Posted - 2012.05.07 13:56:00 -
[26] - Quote
What is "scraping the API" exactly. Second time I've seen this term used and have only heard of it in the context EVE. Searched on the web and couldn't find anything defining this jargon. I'm assuming it means querying an API for all possible values. Is that correct? What does that have to do with "scraping"? Is there are negative connotation to this as scraping doesn't sound particularly pleasant? Extraordinary claims require extraordinary evidence |
EveOrg
EveOrg.Net
2
|
Posted - 2012.05.08 11:12:00 -
[27] - Quote
Khorkrak wrote:What is "scraping the API" exactly. Second time I've seen this term used and have only heard of it in the context EVE. Searched on the web and couldn't find anything defining this jargon. I'm assuming it means querying an API for all possible values. Is that correct? What does that have to do with "scraping"? Is there are negative connotation to this as scraping doesn't sound particularly pleasant?
Heh you are correct, scraping - also known as harvesting - basically getting as much data as possible.
EON App does that. You put Eve api keys into the database, it fills the database with all the API data it can get.
Similar for the market scraper, if you enable the sync you'll get all the market data for all the items in all the regions you want.
Did you have a preferred less 'negative' term for it?!? |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.08 14:46:00 -
[28] - Quote
How about using data mining which is more what you're doing. Scraping is usually used when you are take data from another source then didn't intend the data to be used that way but if you're using people's ApiKeys etc it's not scraping. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Desmont McCallock
172
|
Posted - 2012.05.08 15:09:00 -
[29] - Quote
Scrapping is what we do, through the various libraries we have created for diff programming languages, getting data out of EVE cache files. EVE cache files where created to be used by the EVE client only but we figured out their structure and can now harvest the same data the EVE client uses. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.08 15:13:00 -
[30] - Quote
Grabbing stuff from the cache files would be scraping alright but using ApiKeys to get it from the APis is not which was the main point plus suggesting a better name for that. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
|
|
|
|
Pages: [1] 2 :: one page |
First page | Previous page | Next page | Last page |