Pages: [1] 2 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 11 post(s) |
|
CCP Zirnitra
|
Posted - 2010.08.20 14:02:00 -
[1]
Hi all,
As you may or may not know, the API has been running a bit slow lately and as a result of this we have started an investigation to determine what causes it. One thing we found is that there seems to be quite a few online systems that seem to keep polling API keys that are no longer valid.
Due to this we end up with a lot of unnecessary traffic and load on the API servers, causing degradation in service for everyone, to rectify this we would like all 3rd party developers to ensure that the API keys they are using are still valid. A failed request now and then is no big deal, but when we are talking about some bigger community sites out there we could have hundreds if not thousands of requests per hour from a single system that are using invalid API keys.
If you are developing an application that uses the API, you are responsible for keeping track of the API credentials it uses, meaning that if a key fails multiple times, for example 3-4 times over the cause of a day, you need to disable the API key from being used until it has been updated with a new working key.
To handle the worst cases where we see thousands of bad requests due to bad third party application requests, we will be forced to start putting IP addresses into a blacklist. This means that any request, even ones that would have succeeded, will be blocked until you have fixed your application and requested to be removed. More information on how to check if you have been blacklisted and requesting removal will follow once we are closer to enabling the system.
I hope you all understand the need for this move, and we believe it should help improve the API's performance for everyone.
|
|
Johnathan Walker
Caldari Agent-Orange Nabaal Syndicate
|
Posted - 2010.08.20 14:29:00 -
[2]
\o/ Nice move Zirnitra!
Any idea if the removal request function will go through a petition queue or something separate?
Should be fun going and checking my forum(s) with API authentication now Warmly, "The Bear" JW
|
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2010.08.20 14:34:00 -
[3]
Originally by: CCP Zirnitra I hope you all understand the need for this move, and we believe it should help improve the API's performance for everyone.
Good move, and yes I did have to increase timeouts on API calls recently, nice to hear you are on to it. --
|
Matalino
|
Posted - 2010.08.20 15:12:00 -
[4]
Can we please have a confirmation on best practices interpting the API Error codes?
For error code 200, the supplied Api Key is a limited API Key and no further attempt should be made to use that key for Api's that require a full API Key, as that key will never be a full API key.
For error codes 202, 203, 204, 205, 210, and 212, the supplied Api Key is completely invalid, and that key should be discarded as it cannot be valid in the future.
Error codes 201, 206, 207, 208, 209, 213, 214 are situational. Before repeating an attempt to access these API's, the application should verify that they are accessible by using information in the Account Characters, Character Sheet or Corporation Sheet API's.
Error code 211 indicates that the account currently inactive. The API key may be valid in the future, but attempts to use the key should be limited unless the application recieves an indication that the account has been reactivated.
For inactive accounts, what is the recommended best practice for limiting the frequencey of checking if the account has been reactivated?
|
Dragonaire
Caldari Corax. Circle-Of-Two
|
Posted - 2010.08.20 16:26:00 -
[5]
Safest course would be to deactivate until a person can actually correct or confirm that keys are valid (Account active again) IMHO. If you have lots of people that let their accounts lapse for a few days each month etc. that can become a hassle but if you make it so they can do it themselves it shouldn't be an issue for anyone but them I took that approach with Yapeal when I added deactivating stuff. I also have it just disable APIs as it goes when given limit keys so it doesn't keep try them. It reports errors so application can decide what to do in all cases include if they want to automatically reactivating them which would be a bad idea once CCP does what they are talking about now except if you like being blacklisted -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Trebor Daehdoow
|
Posted - 2010.08.20 16:44:00 -
[6]
A modest suggestion for best practices, if it's not already there: whenever possible, HTTP requests to the API should include a User-Agent: header with contact info for the developer.
Confessions of a Noob Starship Politician Spending Hours blogging the Minutes
|
Dragonaire
Caldari Corax. Circle-Of-Two
|
Posted - 2010.08.20 16:57:00 -
[7]
Something Yapeal also does is add header to requests which application developer can change/ add to. I'll even look at making a config option to add a string to it if there is demand for it if anyone is interested in that use the Yapeal thread I don't mean to pirate a sticky sorry. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Velocity Prime
Misfit Toys
|
Posted - 2010.08.20 19:28:00 -
[8]
This is what CCP is spending their man hours on? The API servers are slow?
Seriously?
We're recruiting! Visit my blog. |
|
Chribba
Otherworld Enterprises Otherworld Empire
|
Posted - 2010.08.20 19:47:00 -
[9]
Sounds like a resonable way of handling it. Is there anyone we can talk to in regards to knowing if we are one of those that are polling a lot of invalid requests?
Of course it would be up to the developer themselves I understand that, why I am asking is because I do know [eb] for instance (from my side) is polling A LOT, and while I do think I have managed to cover all requests and auto disable APIs upon encountering invalid logins I don't want to rule out that I may have missed a line of code somewhere that does not do the needed checks.
And also, while this is nothing I can directly control I have noticed that EVEmon keeps polling inactive accounts every 5 minute regardless of settings (I know I know, delete the inactive accounts from EVEmon... but still) a few thousand EVEmon users with inactive accounts would poll every 5 minutes causing a bit of load I'm sure.
Either way, I would be happy to be aware if I (eveboard) is one of those sites pulling a lot of invalid requests before I accidently end up on a blacklist.
/c
Secure 3rd party service | my in-game channel 'Holy Veldspar' |
|
Squizz Caphinator
|
Posted - 2010.08.20 21:01:00 -
[10]
Depending on how the forum administrator has setup their validation, use of ESAM on large alliance boards could potentially be a problem. As people come and people go, the APIs entered will become invalid. At this time there is no easy way for an admin to clean them up without accessing the database directly. I'll assume the worst, that all forum admins rarely do manual cleanup, and modify ESAM accordingly for future installs.
http://code.google.com/p/esam/issues/detail?id=191 -- Status: Long term hibernation. |
|
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2010.08.21 03:46:00 -
[11]
Can we please see more respectable API error returns, instead of persistent 200-all-well? -- Thanks CCP for cu |
|
CCP Explorer
|
Posted - 2010.08.21 13:45:00 -
[12]
Originally by: Chribba Sounds like a resonable way of handling it. Is there anyone we can talk to in regards to knowing if we are one of those that are polling a lot of invalid requests?
Of course it would be up to the developer themselves I understand that, why I am asking is because I do know [eb] for instance (from my side) is polling A LOT, and while I do think I have managed to cover all requests and auto disable APIs upon encountering invalid logins I don't want to rule out that I may have missed a line of code somewhere that does not do the needed checks.
Either way, I would be happy to be aware if I (eveboard) is one of those sites pulling a lot of invalid requests before I accidently end up on a blacklist.
Chribba, your site is not on our top list of "sites connecting with bad data" that I have seen.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Zirnitra
|
Posted - 2010.08.21 14:13:00 -
[13]
Originally by: Velocity Prime This is what CCP is spending their man hours on? The API servers are slow?
Seriously?
Obviously, not everyone within CCP are able to fix lag/rockets/assault frigates/<insert your personal complaint item here>, so instead of doing nothing while the immensely capable persons who can fix the above work on doing that, other are still working on issues within their area of expertise. As with any other company we have different groups and departments, each responsible for their own part of EVE, and what you are seeing here is a multi-targeted approach at fixing more than a single issue at once.
For information regarding the ongoing efforts lag fixes that we are working incredibly hard on addressing, I would recommend that you read a few of the most recent devblogs from CCP Atropos and CCP Atlas. These are just two of a whole series of devblogs that are already out, with more to come, and they all detail the work that we are currently putting into addressing the lag monster, and help improve the performance for both empire and 0.0 systems.
|
|
Somatic Neuron
|
Posted - 2010.08.21 16:00:00 -
[14]
Edited by: Somatic Neuron on 21/08/2010 16:02:28 So, which Authentication failure errorCodes mean that the API userID/apiKey are invalid (i.e. the apiKey has been changed)? Right now, your errorText leaves a little to be desired.
I don't have a problem removing apiKeys that have been changed, but I am not going to stop polling if the account is disabled, because usually that is a short-time scenario, and we'd want it back as soon as its owner resub. Perhaps if you didn't prevent us from hitting those inactive accounts with the API, you would get fewer errors generated. It isn't like someone can play an account by having access to the API info. ---------- |
|
CCP Stillman
|
Posted - 2010.08.21 19:42:00 -
[15]
Originally by: Somatic Neuron Edited by: Somatic Neuron on 21/08/2010 16:02:28 So, which Authentication failure errorCodes mean that the API userID/apiKey are invalid (i.e. the apiKey has been changed)? Right now, your errorText leaves a little to be desired.
I don't have a problem removing apiKeys that have been changed, but I am not going to stop polling if the account is disabled, because usually that is a short-time scenario, and we'd want it back as soon as its owner resub. Perhaps if you didn't prevent us from hitting those inactive accounts with the API, you would get fewer errors generated. It isn't like someone can play an account by having access to the API info.
Please see the error list for what the difference error codes means.
2xx generally means you should not repeat the same query. To go down the list:
200: Stop 201: Stop 202: Wait some time, the cache on the server could be wrong. But if the problem persists for more than a day or two, stop 203: Stop 204: Stop 205: Stop 206: Unless you have access to change the roles of the character you query, stop 207: Stop 208: Unless you have access to change the roles of the character you query, stop 209: Unless you have access to change the roles of the character you query, stop
1xx also contains some errors which should be a hint to suggest your query is broken. They should be mostly self-explanatory though.
|
|
Vessper
NOVA TECH
|
Posted - 2010.08.21 20:29:00 -
[16]
Originally by: CCP Stillman Please see the error list for what the difference error codes means.
Better yet, see the actual Error API which is a bit more up-to-date EveHQ Character App |
|
CCP Stillman
|
Posted - 2010.08.21 20:46:00 -
[17]
Originally by: Vessper
Originally by: CCP Stillman Please see the error list for what the difference error codes means.
Better yet, see the actual Error API which is a bit more up-to-date
Even better! Thanks!
|
|
Karbowiak
Caldari hirr Morsus Mihi
|
Posted - 2010.08.22 00:32:00 -
[18]
Edited by: Karbowiak on 22/08/2010 00:33:33
Originally by: CCP Explorer
Originally by: Chribba Sounds like a resonable way of handling it. Is there anyone we can talk to in regards to knowing if we are one of those that are polling a lot of invalid requests?
Of course it would be up to the developer themselves I understand that, why I am asking is because I do know [eb] for instance (from my side) is polling A LOT, and while I do think I have managed to cover all requests and auto disable APIs upon encountering invalid logins I don't want to rule out that I may have missed a line of code somewhere that does not do the needed checks.
Either way, I would be happy to be aware if I (eveboard) is one of those sites pulling a lot of invalid requests before I accidently end up on a blacklist.
Chribba, your site is not on our top list of "sites connecting with bad data" that I have seen.
Puh, hope EVE-KILL isn't among the bad ones either (Eventho it surely is, lol - darn EDK) How do you guys btw. feel about public API Proxies? like this: http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1372344 ?
Co-Owner and Creator of EVSCO |
woddel
Gallente Canis Industries Ltd Avaricious Cartel
|
Posted - 2010.08.23 13:27:00 -
[19]
hia zirnitra
thanks for addressing the issue. i also somehow believe that api performance has massively degenerated over the last few weeks or month. of course, this is not your fault and result of ever increasing use of the apis. i also hope that eve commander does not look to bad on your stats. at least as it only polls data on active user request and not on its own.
for informative use, i just created a graph detailing some performance stats measured from my server located in switzerland (based on the serverstatus.aspx which i pull anyway, so not additional traffic is generated because of that).
http://www.eve-commander.com/apiperformance.cfm
regards woddel --- creator and maintainer of eve commander - complete web based character information tool and ec agent finder |
|
CCP Stillman
|
Posted - 2010.08.23 16:19:00 -
[20]
Originally by: woddel hia zirnitra
thanks for addressing the issue. i also somehow believe that api performance has massively degenerated over the last few weeks or month. of course, this is not your fault and result of ever increasing use of the apis. i also hope that eve commander does not look to bad on your stats. at least as it only polls data on active user request and not on its own.
for informative use, i just created a graph detailing some performance stats measured from my server located in switzerland (based on the serverstatus.aspx which i pull anyway, so not additional traffic is generated because of that).
http://www.eve-commander.com/apiperformance.cfm
regards woddel
Very interesting statistics. Thanks for doing this.
|
|
|
Somatic Neuron
|
Posted - 2010.08.24 04:45:00 -
[21]
Originally by: CCP Stillman
Originally by: Somatic Neuron Edited by: Somatic Neuron on 21/08/2010 16:02:28 So, which Authentication failure errorCodes mean that the API userID/apiKey are invalid (i.e. the apiKey has been changed)? Right now, your errorText leaves a little to be desired.
I don't have a problem removing apiKeys that have been changed, but I am not going to stop polling if the account is disabled, because usually that is a short-time scenario, and we'd want it back as soon as its owner resub. Perhaps if you didn't prevent us from hitting those inactive accounts with the API, you would get fewer errors generated. It isn't like someone can play an account by having access to the API info.
Please see the error list for what the difference error codes means.
2xx generally means you should not repeat the same query. To go down the list:
200: Stop 201: Stop 202: Wait some time, the cache on the server could be wrong. But if the problem persists for more than a day or two, stop 203: Stop 204: Stop 205: Stop 206: Unless you have access to change the roles of the character you query, stop 207: Stop 208: Unless you have access to change the roles of the character you query, stop 209: Unless you have access to change the roles of the character you query, stop
1xx also contains some errors which should be a hint to suggest your query is broken. They should be mostly self-explanatory though.
I asked to know which error codes meant that the API key used was no longer valid. RCFTW ---------- |
Lost Hamster
Hamster Holding Corp
|
Posted - 2010.08.24 08:16:00 -
[22]
Originally by: Somatic Neuron
I asked to know which error codes meant that the API key used was no longer valid. RCFTW
<row errorCode="202" errorText="API key authentication failure."/> <row errorCode="203" errorText="Authentication failure."/> <row errorCode="204" errorText="Authentication failure."/> <row errorCode="205" errorText="Authentication failure (final pass)."/> <row errorCode="210" errorText="Authentication failure."/>
So I would say, if you get any of the above one, then, don't query it again.
|
|
CCP Prism X
Gallente C C P C C P Alliance
|
Posted - 2010.08.24 13:13:00 -
[23]
Edited by: CCP Prism X on 24/08/2010 13:15:40 Edited by: CCP Prism X on 24/08/2010 13:15:25 With regards to the authentication errors, there are three different errors cast:
521 Invalid login parameters. Remove immediately from pool. You're passing in something like a negative or null userID or a somehow obviously bad API key.
203 Authentication failure API Key / userID do not match. Remove immediately.
211 Login denied by status User account has lapsed. At least remove this temporarily from the pool if not completely.
Obviously having your IP flagged as someone doing constant failed auth looks very bad but it's not the only errors we'd act on. Any repeated errors coming from your application should be handled as soon as possible. For example we currently have one IP causing 60% of all errors on 3 pages (also, 99% of all errors from those pages) due to improper usage. I'll freely admit that this tells me that the page needs some looking at as it shouldn't be throwing us errors on improper usage to begin with, but it's still not something 3rd party developers should code into effect.
At any rate, we're not implementing these blacklisting procedures to make life difficult for our customers but to increase the reliability of the API which has been somewhat lacking since... yes! I just want to stress that in the case that you get blacklisted we do not intend to keep you that way forever and you wont make anybody's personal ****-list. We'll ensure you have recourse of communication with us to notify us of any progress. It's just one of (hopefully) many new steps to pro-actively maintain the API project for everybody's sake.
P.S. yes I know the error code scheme needs the idiocy beaten out of it as well as the general error handling (as mentioned above) of the API code. It's one of the lots-lots-lots-many-two-one things on the TODO list for the API. ~ Prism X EvE Database Developer Relocating your character to a cozy, secure container since 2006. Relocating your cozy, secure container to the EVE cemetery since 2008. |
|
Lutz Major
Austriae Est Imperare Orbi Universo
|
Posted - 2010.08.24 13:37:00 -
[24]
Originally by: CCP Prism X For example we currently have one IP causing 60% of all errors on 3 pages (also, 99% of all errors from those pages) due to improper usage.
WOAH! Do you know whether there is an acutal application behind this IP or is just someone trying a brute force?
|
|
CCP Prism X
Gallente C C P C C P Alliance
|
Posted - 2010.08.24 13:50:00 -
[25]
Originally by: Lutz Major
Originally by: CCP Prism X For example we currently have one IP causing 60% of all errors on 3 pages (also, 99% of all errors from those pages) due to improper usage.
WOAH! Do you know whether there is an acutal application behind this IP or is just someone trying a brute force?
It's an actual application and the responsible parties have been notified by now.
Probably worth noting that these are NOT autentication errors but actual usage errors (and again they shouldn't be raising any errors to log but that's a different story). ~ Prism X EvE Database Developer Relocating your character to a cozy, secure container since 2006. Relocating your cozy, secure container to the EVE cemetery since 2008. |
|
Arous Drephius
|
Posted - 2010.08.24 14:48:00 -
[26]
Originally by: CCP Prism X It's an actual application and the responsible parties have been notified by now.
Don't suppose you could tell us who (or what page or what they're doing wrong)?
Just curious.
|
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2010.08.24 15:08:00 -
[27]
Don't think they would (either, could) disclose personality behind it, but if there's certain usage patterns that should be avoided (either by now or at all), that would be nice to know, for educational purposes, if not for everyone's benefit. -- Thanks CCP for cu |
manasi
Caldari Ceptacemia
|
Posted - 2010.08.24 15:10:00 -
[28]
Thanks for the info about the API usage.
A friend wrote a web application that checks API's and notifies me if an API is changed (203) or if account is inactive (211) so I do see some of these errors. Is there a way of looking up properly ( that does NOT flag my IP for misuse) that I can ask him to code to. in short is there a best practice for Querying the API info? I did not code the tool but I do use it occasionally, not to mention I am working with two others who are coding a similar tool to use, and if we based this on our first tool, and it is following some bad practice, I wish it to be coded correctly.
Any assistance woudl be appreciated.
Come visit me at http://http://manasi.eveplayer.net |
|
CCP Prism X
Gallente C C P C C P Alliance
|
Posted - 2010.08.24 15:12:00 -
[29]
I could of course.. but I wont. Wouldn't be proper methinks.
"No snitchin'!" ~ Riley Freeman ~ Prism X EvE Database Developer Relocating your character to a cozy, secure container since 2006. Relocating your cozy, secure container to the EVE cemetery since 2008. |
|
Somatic Neuron
|
Posted - 2010.08.24 21:55:00 -
[30]
Originally by: CCP Prism X Edited by: CCP Prism X on 24/08/2010 13:15:40 Edited by: CCP Prism X on 24/08/2010 13:15:25 With regards to the authentication errors, there are three different errors cast:
521 Invalid login parameters. Remove immediately from pool. You're passing in something like a negative or null userID or a somehow obviously bad API key.
203 Authentication failure API Key / userID do not match. Remove immediately.
211 Login denied by status User account has lapsed. At least remove this temporarily from the pool if not completely.
Obviously having your IP flagged as someone doing constant failed auth looks very bad but it's not the only errors we'd act on. Any repeated errors coming from your application should be handled as soon as possible. For example we currently have one IP causing 60% of all errors on 3 pages (also, 99% of all errors from those pages) due to improper usage. I'll freely admit that this tells me that the page needs some looking at as it shouldn't be throwing us errors on improper usage to begin with, but it's still not something 3rd party developers should code into effect.
At any rate, we're not implementing these blacklisting procedures to make life difficult for our customers but to increase the reliability of the API which has been somewhat lacking since... yes! I just want to stress that in the case that you get blacklisted we do not intend to keep you that way forever and you wont make anybody's personal ****-list. We'll ensure you have recourse of communication with us to notify us of any progress. It's just one of (hopefully) many new steps to pro-actively maintain the API project for everybody's sake.
P.S. yes I know the error code scheme needs the idiocy beaten out of it as well as the general error handling (as mentioned above) of the API code. It's one of the lots-lots-lots-many-two-one things on the TODO list for the API.
Thank you, that was exactly what I was looking for. For a 211, is 1x/day to check status okay, or is that still too frequent? ---------- |
|
|
|
|
Pages: [1] 2 :: one page |
First page | Previous page | Next page | Last page |