EVE Search - Authed CREST rate limits

All Channels

EVE Technology Lab

Authed CREST rate limits

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Author	Thread Statistics \| Show CCP posts - 8 post(s)

CCP FoxFour C C P C C P Alliance 3942	Posted - 2015.03.11 09:01:08 - [1] - Quote The actual rate limit is set to something ridiculous, like 100,000 req/s. I want to adjust that to a proper rate limit, but need to decide on what it should be. When I do I will let you all know. I need to figure out why that 503 gets tossed, still not sure as I have not had time to look into it. Sorry. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3970	Posted - 2015.04.08 08:33:45 - [2] - Quote Pete Butcher wrote: Kali Izia wrote: Pete Butcher wrote: For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. 150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone. Is this on pulling market data? @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3970	Posted - 2015.04.08 10:30:26 - [3] - Quote Pete Butcher wrote: CCP FoxFour wrote: Pete Butcher wrote: Kali Izia wrote: Pete Butcher wrote: For crying out loud - this is still happening. Authed CREST throws errors or simply hangs forever despite no official limit being reached. This is really not a good experience for the users, and not a good experience for me trying to explain what's the problem and what to do with it. From what I understand this is related to the proxy issues. Basically each time you make an authed CREST call, any subsequent calls go to the same proxy, and you can end up actually overloading that proxy. To fix it, live market data is being made available on public CREST and the rate limit is being raised to 150 req/s. Those changes are live on Sisi now. 150 is way too little for my needs; I'm sticking with authed version. They should really fix it instead of making workarounds which may not work for everyone. Is this on pulling market data? What kind of rate are you hitting? As an FYI, and this will be explained in an upcoming dev blog, the authed rate limit will be lowered to about 100 req/s, possibly less. Public CREST will be bumped to 150 req/s. Did a little benchmark now and I sent 11364 requests in about 90 milliseconds, which translates to theoretical 126k req/s (in reality i make max couple of tens of thousands total). 100 req/s is quite disconnected from reality in comparison. Evernus users typically download orders and history for major market groups, which is about 20k requests per region. Imagine the time needed to download it all with 100/s. This limit may work for some Web applications with cron running in the background, but it's extremely bad for standalone or real time apps. ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing. This is considered acceptable by us. It's less than the 5 minute cache for the market. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3970	Posted - 2015.04.08 11:00:09 - [4] - Quote Pete Butcher wrote: CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing. This is considered acceptable by us. It's less than the 5 minute cache for the market. You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500? No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3970	Posted - 2015.04.08 11:23:54 - [5] - Quote Pete Butcher wrote: CCP FoxFour wrote: Pete Butcher wrote: CCP FoxFour wrote: ... first of all I am going to highly encourage you to switch to public CREST when the change happens. Which means 150 req/s. 20,000 required requests at 150 per second is 133.3 seconds or 2.2 minutes per region for the whole damn thing. This is considered acceptable by us. It's less than the 5 minute cache for the market. You should add a response time to that. Making requests is just a part of the job. I suspect users will have to wait about 5min per region, which is a lot. Can you increase it to 300-500? No. And I am. Response time while using public CREST is down a fair bit, at least on Sisi and TQ has better hardware. Go ahead and try it your self on Sisi: public-crest-sisi.testeveonline.com I just timed a complete dump of the Tenerifis region using fairly simple and optimized script to pull the whole region (buy and sell orders for every time on the market). It took 3 minutes and 53 seconds. Pretty sure I could get that down with a bit of effort. Ok, we'll see how it's going to perform when it gets live. I hope people won't get too upset, especially those doing region trading between hubs. One last question - what will the response be when the limit gets exceeded? 503 is what you should get when breaking the limit. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3970	Posted - 2015.04.08 12:05:12 - [6] - Quote Pete Butcher wrote: Is 503 possible to be returned in some other circumstances, eg. downtime? I believe downtime gives a 502 or 503 depending on the state of the proxy. Completely offline should be a 502 and starting up should be 503. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 3972	Posted - 2015.04.10 09:11:05 - [7] - Quote Pete Butcher wrote: Ortho Loess wrote: That's the plan. Turned out that authed crest routes all your requests through the same server, so it can't handle having a higher rate. It's on public crest on sisi now, along with the 150/s rate limit. Hmm, that suggests authed CREST has some state stored server side. A bit strange approach since REST is supposed to be stateless to avoid exactly these issues. Very true, however that just the way it has to be due to the EVE server. /welp @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.


CCP FoxFour C C P C C P Alliance 4005	Posted - 2015.04.28 16:00:02 - [8] - Quote Pete Butcher wrote: Any news when this switch to public CREST will take place? There will be a dev blog about it. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.