Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Aximag
|
Posted - 2008.01.14 01:32:00 -
[1]
What cache methods do people use to store API requests?
I have some hacked up python scripts that I run every night that collect xml files for my characters and corporation. These scripts store the results of long cache requests. I then have some hacked up PHP pages that either go to live CCP data for short cache requests or to my local xml files for long cache requests.
There has got to be a better way. This works all right for a small number of characters, but I'd like to generalize my implementation (after significant clean up) for public access.
TIA,
Axi
|
Raquel Smith
Caldari Freedom-Technologies imPure.
|
Posted - 2008.01.14 02:26:00 -
[2]
Originally by: Aximag What cache methods do people use to store API requests?
I have some hacked up python scripts that I run every night that collect xml files for my characters and corporation. These scripts store the results of long cache requests. I then have some hacked up PHP pages that either go to live CCP data for short cache requests or to my local xml files for long cache requests.
There has got to be a better way. This works all right for a small number of characters, but I'd like to generalize my implementation (after significant clean up) for public access.
TIA,
Axi
Reve handles saving of XML and loading of XML from an arbitrary URI (on disk or http).
So that's how I would do it.
-- Creator of The Ruby API Library |
Aximag
|
Posted - 2008.01.14 03:24:00 -
[3]
Thanks for your reply.
Ah yes, Ruby. Another scripting language. I dabbled in Ruby a while back when I wanted to grow trees in Google Sketchup. Not sure I'm savvy enough for that gem. I prefer python, PHP, and bash.
I'll have to look through the code, but I can already load data from an arbitrary source (disk, web, etc.). What I'm looking for is something that is smart enough to cache and fetch within the refresh limits. I'll probably have to code something up myself.
Axi
|
Talaan Stardrifter
|
Posted - 2008.01.14 03:47:00 -
[4]
DIY Caching Proxy =)
Fairly simple. Tricky part would be pulling the 'CachedUntil' out of the returned XML. Once you can grab that, save the raw XML as a file and have the proxy access that until the CachedUntil time expires.
|
Flaming Candle
|
Posted - 2008.01.14 09:25:00 -
[5]
Edited by: Flaming Candle on 14/01/2008 09:28:23 use XPath to get the 'cachedUntil' date
"/eveapi/cachedUntil"
Then you have to convert it to a useful timestamp, which isn't that hard with mktime()/java.util.Calender/etc
As to caching, personally, when I successfully get a result from the API, (not an error) I write the entire file to disk, and throw a record into a database, then when I'm trying to get a file, I check the database first, and return either the cached file, or a fresh version of the API XML.
Abstract the "getXML" method, and have that handle all your caching woes.
|
Ix Forres
Vanguard Frontiers Imperial Republic Of the North
|
Posted - 2008.01.14 21:40:00 -
[6]
Edited by: Ix Forres on 14/01/2008 21:42:03 Edited by: Ix Forres on 14/01/2008 21:40:44 If you do go for Reve, I wrote a caching layer around it which is designed for integrating with Ruby on Rails projects (It's a Rails plugin)- it stores cached data in the database but works transparently (you just use the Recache class rather than Reve). I'll post a link in once I've got my SCM back up- been doing a server move. If you need it urgently drop me an EVEmail and I'll throw it up somewhere that works.
http://www.talkunafraid.co.uk/tags/recache
Basic model is database-based storage of raw cached data (serialized) along with metadata on cache times and so on, plus a hash of the request. You can then do a request to the DB to potentially eliminate an API call.
File based storage is fine to a point but database storage really is the only scalable solution (look ma, buzzwords!)- plus it's a hell of a lot easier to manage and usually quicker.
Blog |
Aximag
|
Posted - 2008.01.15 01:05:00 -
[7]
Ix (Dune?),
Thanks for the help/offer. I'm about half way through doing a cache thing on my XML requests. The code checks the last timestamp on the database copy before going to CCP. I figure I can do this in PHP/MySQL faster than remembering my Ruby.
Best,
Axi
|
Awox
Advanced Logistics
|
Posted - 2008.01.15 04:25:00 -
[8]
I use a slightly modified version of a PHP application, here is the php source code for that file.
You need to create a MySQL table like so:
Quote: CREATE TABLE IF NOT EXISTS `data_cache` (`apicall` text NOT NULL, `key` varchar(64) NOT NULL, `date` int(11) NOT NULL, `value` longtext NOT NULL, KEY `key` (`key`))
(yes I know, reserved keywords as fields is nasty, but I cbf changing..)
Put a section like this in your nginx.conf (or similar rewrite for apache)
Quote: if (!-e $request_filename) { rewrite ^/eveapi/(.*)\.aspx /eveapi/eproxy-rewrite.php break; }
Then, you can access it by simply going to http://yourhost.com/eveapi/eve/AllianceList.xml.aspx instead of http://api.eve-online.com/eve/AllianceList.xml.aspx.
|
Aximag
|
Posted - 2008.01.15 05:13:00 -
[9]
Awox,
This looks right on target -- many thanks. Your implementation looks more refined than mine.
I think you saved me many refactorings.
Axi.
|
Aximag
|
Posted - 2008.01.16 00:30:00 -
[10]
...now if I could only get the mysql PDO driver installed on my OS X 10.5 box...
|
|
Isidien Madcap
GoonFleet GoonSwarm
|
Posted - 2008.01.17 00:26:00 -
[11]
Note that I updated the original eproxy a week or so ago to remove the PDO dependency, to have it create the data tables if they don't exist, and a few other fixes. Should work with the straight non-PDO mysql driver (as well as a PDO one). ---- Isidien |
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |