EVE Search - API and market data to SQL, and API Proxy, in one tool

All Channels

EVE Technology Lab

API and market data to SQL, and API Proxy, in one tool

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Author	Thread Statistics \| Show CCP posts - 0 post(s)
EveOrg EveOrg.Net 0	Posted - 2011.11.01 09:53:00 - [1] - Quote EveOrg App has been updated (finally) to use the new API key system. Other changes include... * no registration required, fully free to use personally, corp or alliance wide * improved API data processing, now more efficient and reliable * API data scraping expanded to encompass all available API data (except server status) The EOApp runs under .Net (v3.5) on Windows (AFAIK it does not work on mono). It does three things - in a fancy multi-threaded way ... API 2DB You provide a SQL database and put some API keys in it. API2DB will continually scrape the API for all available data for all the keys and keep the SQL database up to date with the latest data 24/7. Some data (like market orders and sovereignty) also build historically to provide a useful data mine. Every piece of data available from the API is scraped. Assets, Contracts, Market Orders, Notifications, Research + Industry Jobs, Contacts, Standings, Main, lock stock the f@#!ing lot. All public data is scraped, sovereignty, map kills/jumps, alliance list, etc. Even useful anonymous data is scraped about characters and corps - e.g. your wallet transaction data reports selling an item to character "EveOrg" then as much data about EveOrg as possible is scraped from public API calls (like CharacterInfo). What can you use this for? What can't you use it for! Monitor and plan your trading. Gather intel on corp applicants. Alliance wide detection of hostile activities. Backend to a website. Monitor corp contracts. Easily queryable security logs. Anything you can dream up to do with the data is just a SQL query away :) Market 2DB Monitors the game client cache folder for market data. As soon as a new market cache file appears the data is added to a table in the SQL database. You can rapidly build up a database of accurate up to date market orders/prices/etc by clicking through the market in-game. API Proxy Acts as a proxy for the API. Point your other API consuming applications at it instead of CCP and always get the latest data (rather than the API rejecting apparently repeated calls). It's pretty much at late beta stage. Thanks to some early beta testers from a thread in the old forums it now fairly reliable and efficient So I'm making it available for public consumption. Free of charge. No registration. Donations of isk are of course appreciated Please report bugs in this thread, or to me via Eve mail, of drop into the in-game channel eveorg.net. Hope people find it useful!
EveOrg EveOrg.Net 0	Posted - 2011.11.01 09:56:00 - [2] - Quote I would also be interested in hearing from people wanting to integrate other tools/services. Like centralized market data etc.
EveOrg EveOrg.Net 0	Posted - 2011.11.01 22:05:00 - [3] - Quote The connection is over ODBC, you give it an ODBC connection string. It talks SQL to the DB - it's expecting an MS SQL DB not sure how it would fair with other DBs. Please try and let me know :) It works fine with the free version of MS SQL Express if it's a cost thing.
EveOrg EveOrg.Net 0	Posted - 2011.11.02 00:46:00 - [4] - Quote None of this any help? MySQL connection strings Or shoot me an evemail with details of the server if it's publically accessible I'll see what I can do. I'll spend some time this weekend setting up mySQL on a linux VM and see if I can make it work(tm).
EveOrg EveOrg.Net 0	Posted - 2011.11.02 01:06:00 - [5] - Quote No need for a full license, SQL Express is up to the task and completely free. I will have a go at making it work with MySQL, if nothing else the more options people have for deployment the more useful it is - and the more chance of generous isk donations being heaped upon me Plus I've been meaning to try this linux thing for a while...
EveOrg EveOrg.Net 0	Posted - 2011.11.05 02:36:00 - [6] - Quote Jenn Makanen wrote: Have you thought about using odbc for the database connection? (Disclaimer: I have no idea how hard this would be) I've been playing with the python scrapers to get stuff into a remote mysql DB, and this tool looks like it would handle it better. New version of EveOrg App out - supporting MySQL :) Now you can configure for MSSQL or MySQL compatibility. All your API data in the database of your choice! Also fixed a few bugs and that EveOrg App
EveOrg EveOrg.Net 1	Posted - 2012.03.18 11:02:00 - [7] - Quote Hi, sorry been a while since last update. Had to go away for a month abroad to work. Back and with an update to EveOrg. Will attempt to answer some questions posted above... dumper.exe crashing. I found a new version of dumper from the libevecache website fixed that (something changed!) Kepuh Harjo, having to recapitalise all the fieldnames in code would be a big job, maybe next update sorry. 6ie, fixed 'cacheTS' bug in this version. 6ie, Nathan Gorden, fixed dateadd bug. Sorry you had to go to the trouble of installing ms-sql ;) Sorry about the lowecase/camelcase problems. I will address them in a future update.
EveOrg EveOrg.Net 1	Posted - 2012.05.07 06:21:00 - [8] - Quote Yes, more than likely the DB changes. Giant new update has been released that takes care of that, and some other things. Including.. Totally Refactored Market Sync Massively more scalable and efficient market sync feature. Sync your market price database with the central server and get universe wide price data (assuming someone has uploaded it). Yes, a bit like Eve-Central et al. This central price data is also available to anyone who wants to query it with some code/script, instructions on the Market Prices page. Or you can just let the app do the work then query your own DB yourself. I just noticed a forum thread about combining market data from the other market data websites, guess I'll have a read of that and see what's going on there...
EveOrg EveOrg.Net 2	Posted - 2012.05.08 11:12:00 - [9] - Quote Khorkrak wrote: What is "scraping the API" exactly. Second time I've seen this term used and have only heard of it in the context EVE. Searched on the web and couldn't find anything defining this jargon. I'm assuming it means querying an API for all possible values. Is that correct? What does that have to do with "scraping"? Is there are negative connotation to this as scraping doesn't sound particularly pleasant? Heh you are correct, scraping - also known as harvesting - basically getting as much data as possible. EON App does that. You put Eve api keys into the database, it fills the database with all the API data it can get. Similar for the market scraper, if you enable the sync you'll get all the market data for all the items in all the regions you want. Did you have a preferred less 'negative' term for it?!?
EveOrg EveOrg.Net 2	Posted - 2012.05.08 20:52:00 - [10] - Quote Dragonaire wrote: How about using data mining which is more what you're doing. Scraping is usually used when you are take data from another source then didn't intend the data to be used that way but if you're using people's ApiKeys etc it's not scraping. Data mining is what you do with the data once it's in the database. Scraping is the process of collecting the data. I think we're getting a bit semantic here ;) How about API spider, or crawler, or vacuum?

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.