| Pages: [1] :: one page |
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
BigSako
Burning Napalm
Northern Coalition.
84
   |
Posted - 2013.08.22 10:23:00 -
[1] - Quote
While developing I have discovered a problem that most developers will come across.
I have a cronjob that runs regularly to check some APIs, and the question is:
Should I make this cronjob multi-threaded?
Our server has enough capabilities to run it multi-threaded, the database and the used libraries also support multi-threading.
The question is: Will the API-Server be happy with it?
It would work like this:
Get list of API KeyId and VCODE
Split it in 4 chunks
Process the 4 chunks.
Each API-process would access several API information like APIKeyInfo and CharacerSheet, and more.
Getting the data from one API Key takes about 5.2 seconds at the moment, with the most time being spent on actually calling the API and receiving the data, not the actual database processing on our server.
We're talking about 300+ API Keys to process here, so the cronjob would take like 260 minutes (OMG).
I know I can split the load over time, but even if I run only like 60 API Keys at a time we are still speaking about a 5 minute lasting cronjob, with most of the time spend idle because of the https requests.
So back to my question: Will the API server be happy if I multi-thread the API requests (e.g. 4 threads), or would that be too many requests at once?
PS: Caching times are being followed, I don't run this job every 5 minutes... |
ItsmeHcK1
Kicked.
Shadow Cartel
80
|
Posted - 2013.08.22 21:15:00 -
[2] - Quote
I multithread calls in my API checker and haven't heard any complaints. Don't think it reaches your scale though. Best I've heard someone do is 100 checks at the same time, coming to about 2000-3000 calls or so in 5 minutes, but that's a one-time thing, not a daily occurrence. |
BigSako
Burning Napalm
Northern Coalition.
84
|
Posted - 2013.08.23 14:17:00 -
[3] - Quote
well, the amount of calls per HOUR would probably be around 3000 and that wouldn't change whether it is threaded or not.
edit: but let's say I go after the asset list and/or wallet at some point, then I would probably do that max. once per day for each API.
But when doing that I wouldn't like the cronjob to be busy for 10 minutes or so. |
| |
|
| Pages: [1] :: one page |
| First page | Previous page | Next page | Last page |