Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 1 post(s) |
Parkeur
Science and Trade Institute Caldari State
0
|
Posted - 2015.03.07 00:38:24 -
[1] - Quote
For those of you who have written a Market Orders downloader using the CREST market orders endpoint....
How long does it take your app to download all buy and sell orders for all market types for The Forge region and store the data somewhere? I'm asking because it seems it would take many hours to download all orders for all regions and I wonder if others are doing it radically quicker than I am.
I'll risk embarrassing myself first:
20 minutes (9189 market types, 228,218 orders).
The main components are:
Flow is implemented in Spring Integration XML with some Java for messaging endpoints Jackson for JSON handling Google oauth2 library Data stored in HSQLDB 4 threads pulling data from CREST, rate limited at 25 per second
|
Pete Butcher
KarmaFleet Goonswarm Federation
276
|
Posted - 2015.03.07 07:18:04 -
[2] - Quote
Parkeur wrote:For those of you who have written a Market Orders downloader using the CREST market orders endpoint.... How long does it take your app to download all buy and sell orders for all market types for The Forge region and store the data somewhere? I'm asking because it seems it would take many hours to download all orders for all regions and I wonder if others are doing it radically quicker than I am. I'll risk embarrassing myself first: 20 minutes (9189 market types, 228,218 orders). The main components are: Flow is implemented in Spring Integration XML with some Java for messaging endpoints Jackson for JSON handling Google oauth2 library Data stored in HSQLDB 4 threads pulling data from CREST, rate limited at 25 per second
To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. CREST is really slow if you want to do mass import. Most of the time your threads will be IO bound, so you can make a lot of them, especially with HT. Benchmark and find your optimal number. Of course, after downloading it all, your bottleneck would shift to the storage, but if you want to do some in memory processing, you'll be fine.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Kali Izia
GoomWaffe Goonswarm Federation
36
|
Posted - 2015.03.08 05:36:51 -
[3] - Quote
Pete Butcher wrote:To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. Wait what
You don't need to use multiple clients IDs. The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can.
CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about. |
Pete Butcher
KarmaFleet Goonswarm Federation
276
|
Posted - 2015.03.08 07:43:32 -
[4] - Quote
Kali Izia wrote:Pete Butcher wrote:To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. Wait what You don't need to use multiple clients IDs. The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can. CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.
As far as I know, there is a limit on the number of requests an app can make. Can't remember what it is tho. Do you have any info on that?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Kali Izia
GoomWaffe Goonswarm Federation
36
|
Posted - 2015.03.08 08:02:26 -
[5] - Quote
There isn't. There are rate limits for public CREST, which you should adhere to for all non-market endpoints on authed CREST, but otherwise there are no other hard limits. |
Pete Butcher
KarmaFleet Goonswarm Federation
276
|
Posted - 2015.03.08 08:15:41 -
[6] - Quote
Kali Izia wrote:There isn't. There are rate limits for public CREST, which you should adhere to for all non-market endpoints on authed CREST, but otherwise there are no other hard limits.
Oh, if that's the case, you can indeed use one app. My bad.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3941
|
Posted - 2015.03.09 10:26:23 -
[7] - Quote
Kali Izia wrote:Pete Butcher wrote:To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. Wait what You don't need to use multiple clients IDs. The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can. CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.
Very much this. :D
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Parkeur
Science and Trade Institute Caldari State
0
|
Posted - 2015.03.10 00:40:47 -
[8] - Quote
Kali Izia wrote:Pete Butcher wrote:To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. Wait what You don't need to use multiple clients IDs. The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can. CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about.
Thanks for this Kali Izia, I've done some tweaking on the database, a bit of code optimization, removed the rate limiter and upped the threadpool size and it now takes 6 minutes for all buy/sell orders for The Forge. For anyone that's interested:
- 10,513 market items
- 9193 actually have orders in The Forge
- 229,699 total orders
- 25 Java threads
- 58 HTTP requests per second
- 638 database rows per second inserted
My 5-year old laptop was indeed breaking a sweat at this point so I don't think it will go much higher on my hardware but nevertheless it's not bad for a |
Pete Butcher
KarmaFleet Goonswarm Federation
276
|
Posted - 2015.03.10 08:13:45 -
[9] - Quote
CCP FoxFour wrote:Kali Izia wrote:Pete Butcher wrote:To speed things up, create a ton of separate apps on the dev portal and pull concurrently with full speed. Wait what You don't need to use multiple clients IDs. The second part of that sentence is accurate though. Remove any rate limiting that you're doing (for the market order endpoint only, you should still implement rate limiting for all others) and hit the server as fast and as hard as you can. CCP does technically have a rate limit, but it's high enough to allow you to pull the entire universe's market within the 5 minute cache period without breaking a sweat, so it's not something you realistically need to worry about. Very much this. :D
Are sure there's no limit? I just fired up fetching market history at full speed (hundreds, if not thousands, of requests per second) and got a crapton of "Bad gateway" errors.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |