Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 3 post(s) |
Ethan02
Succurit Research and Development
0
|
Posted - 2014.05.12 04:02:00 -
[1] - Quote
Good morning,
found out about the CREST endpoint for market data history this weekend and played around with it. Goal would be to mirror/import the data into my own database and have them then analyzed for trends.
If I see this correctly, then there are 10,377 tradeable types and 97 regions. Thus in order the "get it all" I would expect to make 1,006,569 queries against CREST. That is a lot and may put some load on the CCP server.
This put this into perspective, if I would do this in serial and allow a second of delay between the requests, then the whole transfer would take more than 11 days.
AM I missing something? Is there a way to get more data (all regions) within one request? Is it allowed to hammer the CREST API like that? If not, what is the recommended way to do this.
At best I'd like to have the data on a weekly basis (then obviously only the last week is of interest).
Thanks for the help. |
Gordon Newman
Deep Core Mining Inc. Caldari State
8
|
Posted - 2014.05.12 07:54:00 -
[2] - Quote
Yeah that's a problem. Although you could use multiple parallel connections (the CREST API should be able to handle that load), there might not be relevant history data for every region/type combination, so that you'd get a lot of 'typeNotOnMarket' errors. I'm currently working on feeding CREST data into EMDR, so you only get history updates for active region/type combinations once every 24h. All you would have to do is to filter the EMDR feed for data uploaded by our consumer. That would require you to listen to the EMDR feed 24/7 to catch all updates.
Alternatively we could add an API/daily data dump for that to Element43, as we already have all historic data in our DB. Co-Founder of element43 - Join our team! |
|
CCP FoxFour
C C P C C P Alliance
3222
|
Posted - 2014.05.13 17:37:00 -
[3] - Quote
Slowley... but really why do you need all of this data from all of the regions?
Generally speaking please keep public CREST requests down to 10 requests per seconds. Then yes, go through all of those requests. CCP FoxFour // Game Designer // @regnerba
|
|
Scarna Jenari
New Eden Technology Services
0
|
Posted - 2014.05.13 18:14:00 -
[4] - Quote
30 regions are WH space so there's no real point in querying those.
|
brammator
SUB ZERO. Legion of xXDEATHXx
97
|
Posted - 2014.06.19 10:56:00 -
[5] - Quote
Scarna Jenari wrote:30 regions are WH space so there's no real point in querying those.
That's a real relief, now we could download all daily data in about 20 hours! Yay! |
brammator
SUB ZERO. Legion of xXDEATHXx
97
|
Posted - 2014.06.19 10:59:00 -
[6] - Quote
CCP FoxFour wrote:Slowley... but really why do you need all of this data from all of the regions?
Well, for example I would like to search for best trade routes. That definitely need data about all regions.
|
|
CCP FoxFour
C C P C C P Alliance
3301
|
Posted - 2014.06.19 12:41:00 -
[7] - Quote
brammator wrote:CCP FoxFour wrote:Slowley... but really why do you need all of this data from all of the regions?
Well, for example I would like to search for best trade routes. That definitely need data about all regions.
OK, well at this time there are no plans to offer a get multiple option. So either slim the number of regions/types you are asking about or wait 17 hours for a full refresh. CCP FoxFour // Game Designer // @regnerba
|
|
|
CCP FoxFour
C C P C C P Alliance
3301
|
Posted - 2014.06.19 13:32:00 -
[8] - Quote
Also please ensure you are checking your type has a marketGroupID before passing it to the endpoint. The SDE contains that information in the column marketGroupID.
You should also be able to only do items with a typeID < 350000 as everything above is DUST. CCP FoxFour // Game Designer // @regnerba
|
|
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |