Pages: [1] 2 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 8 post(s) |
Pete Butcher
KarmaFleet Goonswarm Federation
276
|
Posted - 2015.03.10 20:48:17 -
[1] - Quote
Since we have a public CREST rate limits thread, let's make this one also visible to the developers without the need to search. So, let's dig into the problem. Here I got info that we should not care about limits for authed CREST. I followed that advice and pulled market data+history for < 10k items. I didn't actually measure the requests/s, since I was told not to care, but they all got send in a few s. At some point, the CREST server started throwing "Bad gateway" errors and eventually stopped responding at all. By lowering the rate, everything went fine. Therefore the official question is:
Is there or is there not a rate limit for authed CREST, and what it is if it exists (pretty sure it does)?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Ortho Loess
Pentag Blade Curatores Veritatis Alliance
55
|
Posted - 2015.03.10 21:15:01 -
[2] - Quote
I asked back in November.
The answer was that it's still being decided.
FoxFour: maybe you have enough data to have set it now? |
Pete Butcher
KarmaFleet Goonswarm Federation
277
|
Posted - 2015.03.10 21:25:56 -
[3] - Quote
Whatever it is, I can already say a standalone app on a typical desktop PC can easily reach it. And I really, really hope it's not on per-app basis.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Kali Izia
GoomWaffe Goonswarm Federation
39
|
Posted - 2015.03.11 06:25:02 -
[4] - Quote
AFAIK you should adhere to the public rate limit for market history, even on authed CREST. It's just market orders you're allowed to hammer. That might be contributing to the errors if it's rate limit related.
For what it's worth, I tried to reproduce this issue and I was only able to get errors when I set my max sockets to a stupidly high number (10000). In that case I was getting 502s with "Gateway error communicating with EVE proxy". I think that might actually just be overloading CREST. I believe rate limits are supposed to throw a 503 instead.
I'm sure FoxFour will be able to answer though. |
Pete Butcher
KarmaFleet Goonswarm Federation
277
|
Posted - 2015.03.11 07:06:11 -
[5] - Quote
Kali Izia wrote:AFAIK you should adhere to the public rate limit for market history, even on authed CREST. It's just market orders you're allowed to hammer. That might be contributing to the errors if it's rate limit related.
I hope that's not the case, since it would be pretty damn stupid. Either we're using authed CREST with all its benefits or not. There's no middle ground here. But, on the other hand, we have authed urls in public CREST which violates both common sense and HATEOAS, which crest seems trying to adhere to.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3942
|
Posted - 2015.03.11 09:01:08 -
[6] - Quote
The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
277
|
Posted - 2015.03.11 09:06:02 -
[7] - Quote
CCP FoxFour wrote:The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry.
If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Pete Butcher
KarmaFleet Goonswarm Federation
277
|
Posted - 2015.03.13 17:31:08 -
[8] - Quote
Any info on that one? It's still happening.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Ereshgikal
Strange Energy The Bastion
47
|
Posted - 2015.03.16 18:36:38 -
[9] - Quote
Pete Butcher wrote:CCP FoxFour wrote:The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry. If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing.
Just a random idea: Could it be your ISP (or someone else on the network between you and CCP) that considers your traffic harmful and blockholes it for a while? Given the behavior of some ISPs this wouldn't surprise me. Though that wouldn't explain the 503 unless the ISP injects itself into your SSL stream... :( But it might explain the loss of contact later. |
Pete Butcher
KarmaFleet Goonswarm Federation
279
|
Posted - 2015.03.16 18:48:26 -
[10] - Quote
Ereshgikal wrote:Pete Butcher wrote:CCP FoxFour wrote:The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry. If you want to see what's causing this, I can fire up the import at any time. The gateway errors are reproducible. What's more disturbing is the total lack of any response after some time. The server just seems to hang in there doing nothing. Just a random idea: Could it be your ISP (or someone else on the network between you and CCP) that considers your traffic harmful and blockholes it for a while? Given the behavior of some ISPs this wouldn't surprise me. Though that wouldn't explain the 503 unless the ISP injects itself into your SSL stream... :( But it might explain the loss of contact later.
No, it's crest. You just need to fire a lot of requests fast and it will die despite never reaching the official limit. Well, I'll be happy to actually see it die in a deterministic fashion, but it sometimes swallows requests without responding and my application just sits there wondering wtf.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
281
|
Posted - 2015.04.07 18:08:25 -
[11] - Quote
For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and a good experience for me trying to explain what's the problem and what to do with it.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
Kali Izia
GoomWaffe Goonswarm Federation
41
|
Posted - 2015.04.07 21:30:27 -
[12] - Quote
Pete Butcher wrote:For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy.
To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. |
Pete Butcher
KarmaFleet Goonswarm Federation
285
|
Posted - 2015.04.08 04:32:29 -
[13] - Quote
Kali Izia wrote:Pete Butcher wrote:For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now.
150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3970
|
Posted - 2015.04.08 08:33:45 -
[14] - Quote
Pete Butcher wrote:Kali Izia wrote:Pete Butcher wrote:For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. 150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone.
Is this on pulling market data?
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
286
|
Posted - 2015.04.08 09:32:14 -
[15] - Quote
CCP FoxFour wrote:Pete Butcher wrote:Kali Izia wrote:Pete Butcher wrote:For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. 150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone. Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s.
Did a little benchmark now and I sent 11364 requests in about 90 milliseconds, which translates to theoretical 126k req/s (in reality i make max couple of tens of thousands total). 100 req/s is quite disconnected from reality in comparison. Evernus users typically download orders and history for major market groups, which is about 20k requests per region. Imagine the time needed to download it all with 100/s. This limit may work for some Web applications with cron running in the background, but it's extremely bad for standalone or real time apps.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3970
|
Posted - 2015.04.08 10:30:26 -
[16] - Quote
Pete Butcher wrote:CCP FoxFour wrote:Pete Butcher wrote:Kali Izia wrote:Pete Butcher wrote:For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. 150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone. Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s. Did a little benchmark now and I sent 11364 requests in about 90 milliseconds, which translates to theoretical 126k req/s (in reality i make max couple of tens of thousands total). 100 req/s is quite disconnected from reality in comparison. Evernus users typically download orders and history for major market groups, which is about 20k requests per region. Imagine the time needed to download it all with 100/s. This limit may work for some Web applications with cron running in the background, but it's extremely bad for standalone or real time apps.
... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.
This is considered acceptable by us. It's less than the 5 minute cache for the market.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
286
|
Posted - 2015.04.08 10:51:39 -
[17] - Quote
CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.
This is considered acceptable by us. It's less than the 5 minute cache for the market.
You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3970
|
Posted - 2015.04.08 11:00:09 -
[18] - Quote
Pete Butcher wrote:CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.
This is considered acceptable by us. It's less than the 5 minute cache for the market.
You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500?
No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com
I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
286
|
Posted - 2015.04.08 11:16:40 -
[19] - Quote
CCP FoxFour wrote:Pete Butcher wrote:CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.
This is considered acceptable by us. It's less than the 5 minute cache for the market.
You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500? No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort.
Ok, we'll see how it's going to perform when it gets live. I hope people won't get too upset, especially those doing region trading between hubs.
One last question - what will the response be when the limit gets exceeded?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3970
|
Posted - 2015.04.08 11:23:54 -
[20] - Quote
Pete Butcher wrote:CCP FoxFour wrote:Pete Butcher wrote:CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing.
This is considered acceptable by us. It's less than the 5 minute cache for the market.
You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500? No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort. Ok, we'll see how it's going to perform when it gets live. I hope people won't get too upset, especially those doing region trading between hubs. One last question - what will the response be when the limit gets exceeded?
503 is what you should get when breaking the limit.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
286
|
Posted - 2015.04.08 11:34:55 -
[21] - Quote
Is 503 possible to be returned in some other circumstances, eg. downtime?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3970
|
Posted - 2015.04.08 12:05:12 -
[22] - Quote
Pete Butcher wrote:Is 503 possible to be returned in some other circumstances, eg. downtime?
I believe downtime gives a 502 or 503 depending on the state of the proxy. Completely offline should be a 502 and starting up should be 503.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Aineko Macx
345
|
Posted - 2015.04.09 08:01:33 -
[23] - Quote
CCP FoxFour wrote:As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s. Are you moving the market orders endpoint over to public CREST? Otherwise having authed CREST more restricted makes no sense at all.
iveeCrest: A PHP library for CREST || iveeCore: The PHP engine for industrial activities
|
Ortho Loess
Pentag Blade Curatores Veritatis Alliance
56
|
Posted - 2015.04.09 21:06:08 -
[24] - Quote
That's the plan.
Turned out that authed crest routes all your requests through the same server, so it can't handle having a higher rate.
It's on public crest on sisi now, along with the 150/s rate limit. |
Pete Butcher
KarmaFleet Goonswarm Federation
286
|
Posted - 2015.04.10 05:41:08 -
[25] - Quote
Ortho Loess wrote:That's the plan.
Turned out that authed crest routes all your requests through the same server, so it can't handle having a higher rate.
It's on public crest on sisi now, along with the 150/s rate limit.
Hmm, that suggests authed CREST has some state stored server side. A bit strange approach since REST is supposed to be stateless to avoid exactly these issues.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
3972
|
Posted - 2015.04.10 09:11:05 -
[26] - Quote
Pete Butcher wrote:Ortho Loess wrote:That's the plan.
Turned out that authed crest routes all your requests through the same server, so it can't handle having a higher rate.
It's on public crest on sisi now, along with the 150/s rate limit. Hmm, that suggests authed CREST has some state stored server side. A bit strange approach since REST is supposed to be stateless to avoid exactly these issues.
Very true, however that just the way it has to be due to the EVE server. /welp
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Pete Butcher
KarmaFleet Goonswarm Federation
289
|
Posted - 2015.04.28 15:05:17 -
[27] - Quote
Any news when this switch to public CREST will take place?
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
CCP FoxFour
C C P C C P Alliance
4005
|
Posted - 2015.04.28 16:00:02 -
[28] - Quote
Pete Butcher wrote:Any news when this switch to public CREST will take place?
There will be a dev blog about it.
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Gordon Newman
Deep Core Mining Inc. Caldari State
13
|
Posted - 2015.06.21 14:55:27 -
[29] - Quote
I'm sorry for bringing this up again - is there any progress? I'm running into the same issues while building a new backend for EMDR (CREST-based instead of cache scraping) and I can't get stable connectivity at over 100 requests/second. I always get those EVE Proxy errors for ~50% of my requests which is unsustainable. After analysing Element43's database I figured that there are around 380k markets (region/type combinations) with orders in the market. If we were to fetch those every 5 minutes (x2 because ask and bid are two different endpoints) and fetch markets without any orders at a reduced rate like every 60 minutes, we'd still end up with way over 3k requests per second. Again - over three thousand OAuth'ed SSL-Encrypted requests per second. With some clever software engineering this might be achievable with a single server, however at the moment CREST is barely able to deliver a 30th of the needed volume. Maybe we'd all be better off with an aggregated market data endpoint e.g. all orders per type or region. This would greatly reduce the number of requests and the unnecessary overhead they generate, even with connection pooling. Or, even better, an official market data stream.
Co-Founder of element43 - Join our team! |
EVE Live | EMDRMon | accounts-eve
|
Pete Butcher
KarmaFleet Goonswarm Federation
293
|
Posted - 2015.06.21 20:50:36 -
[30] - Quote
Gordon Newman wrote:Maybe we'd all be better off with an aggregated market data endpoint e.g. all orders per type or region. This would greatly reduce the number of requests and the unnecessary overhead they generate, even with connection pooling. Or, even better, an official market data stream.
^ this. Getting stable market replies is a lottery now. I bet a keg of beer even those 100r/s will be too much if enough applications start polling the data via the same proxy. Again - please give us a simple versioned WADL along with (instead of?) HATEOAS and everyone will benefit from it - less work for us and less load for servers.
http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool + Trade Advisor
|
|
|
|
|
Pages: [1] 2 :: one page |
First page | Previous page | Next page | Last page |