EVE Search - The transaction/journal API limit

Search EVE-Online forums for:

All Channels

Market Discussions

The transaction/journal API limit

Monitor this thread via RSS [?]

Pages: [1] :: one page

Author	Topic
Ambo Dirty Deeds Corp. Axiom Empire	Posted - 2008.07.04 08:01:00 - [1] After reading Shar's post, I thought this problem deserved a thread of it's own so as not to take over the other one. Originally by: Shar Tegral So, what troubles me is why one of the common standards we all believe in is not proving to be true. 1000 records or, if more than 1000 records in a week, one week's worth of records (no matter how many). From my testing this no longer holds true. Well we already know that every now and then, CCP will clear out the older data otherwise they would run out of ID numbers. It may be that with the most recent clearout they just got rid of more data than normal. What we need to find out is: 1) Is there a common cutoff point? (In terms of either number of transactions or date) 2) Does the cutoff point move over days/weeks? In other words, have the rules completely changed or is this just a temporary thing due to database maintenance on CCP's part? -------------------------------------- Trader? Investor? Just want to track your finances? Check out EMMA
Ambo Dirty Deeds Corp. Axiom Empire	Posted - 2008.07.04 08:05:00 - [2] Here are my results. (I've been very inactive in the market lately so not incredibly helpful) Transactions total: 27 oldest: 23rd June Journal total: 1000 oldest 8th April -------------------------------------- Trader? Investor? Just want to track your finances? Check out EMMA
Shar Tegral	Posted - 2008.07.04 14:16:00 - [3] Originally by: Ambo I've been very inactive in the market lately so not incredibly helpful But that is why your results are very helpful. Since you've not been doing the market much you should have 1000 records going back further than one week. This is not the case so obviously there is some "other" over-riding standard. Hey you, the dev that reads MD, could you find out for us about this please? Inference on technical matters only feeds the aluminum beanie crowd and frustrates the rest of us. (Yeah, I know. That implies I'm not a beanie wearer!) To Shar -verb: 1 - To say what you mean. 2 - To say what it means. 3 - To say something mean.
Mr Horizontal Gallente KIA Corp KIA Alliance	Posted - 2008.07.04 14:32:00 - [4] Well according to eve-dev wiki: Quote: Journal Walking The journal is configured to provide up to 1000 entries per request. It is also configured to allow you to view up to one week's worth of data. If the first 1000 entries you view stretch back further than a week, then you will not need to do any walking. However, if you have enough entries in your journal that 1000 is less than a week, you can walk backwards in the log to get more data. Walking is performed by specifying the beforeRefID parameter as the lowest refID you have seen so far. You will then get another page with entries. This process can be repeated until you have gone back a week. The server will return an error if you attempt to go back farther. Programmatically, it will be easiest to just loop while you are getting the max number of entries from the server, stopping when the server returns less than the max (i.e. you get 853 entries in a pull), the oldest entry is over a week old, or the server returns an error. I've highlighted the work you have to do to 'know' where the journal has ended at. Director \| www.eve-bank.net
Shar Tegral	Posted - 2008.07.04 15:13:00 - [5] Originally by: Mr Horizontal I've highlighted the work you have to do to 'know' where the journal has ended at. So what you are saying is that if I only get 93 entries I need to start stepping the records back for more? I didn't think so mate. We are not discussing our inability to step back record gathering at all. We are discussing the fact that api's that should have 1000 records, minimum, do not have access to those 1000. They simply are not there. To Shar -verb: 1 - To say what you mean. 2 - To say what it means. 3 - To say something mean.
Starlight Horizons Starlight Horizons Corporation New Eden Research	Posted - 2008.07.05 02:22:00 - [6] This has been something that has been bothering me for a while now. For probably 95% of all of Eve they don't care, but I had more than 9,000 transactions last month. I download my wallet transactions everyday before work and at least once or twice before I go to bed. I run my own macro's in excel, I have used Eve Meep and a few other programs to track my transactions but I like running several different reports, charts, etc and I like having control of all of my data. I was also subjected to API warfare as someone thought it would be funny to sell me trit one unit at a time 435 times. They wasted alot of time and caused unnecessary database load. So, I will be on the look out for that guy this weekend IPO Blog
Rho'varo Diversified Operational Services	Posted - 2008.07.05 03:31:00 - [7] Originally by: Starlight Horizons I was also subjected to API warfare as someone thought it would be funny to sell me trit one unit at a time 435 times. Maybe he just had a tough mission and lost 435 rookie ships in a row? Features & Ideas: Winding Up Learning Skills
Starlight Horizons Starlight Horizons Corporation New Eden Research	Posted - 2008.07.05 03:36:00 - [8] If he has figured out how to undock, warp in, get killed, get new ship, sell trit and do it all again in under 15 seconds then I want to hook up to his internet provider. They were all one after another, I had no other orders come in or out during the time span. Of course, maybe it is a bug and I should petition, maybe he tried to sell me trit and he was lagging and kept hitting the sell button over and over again while screaming at his computer to f*ckin work! IPO Blog
Ray McCormack hirr	Posted - 2008.07.05 11:17:00 - [9] I think it goes something like this : 1) One week. 2) Some random time-span (if less than 1000 transactions). 3) One thousand transactions. If you do one transaction a day, you won't be getting three year's worth of data back, there's still an additional cut-off. I don't think this is a static cut-off though. But, no matter what, you should always be able to get at least one week's worth of data. [WTS] Tech II Component BPO Sets
Lady Centrina	Posted - 2008.07.23 01:48:00 - [10] Ok, bit of a bump here but I am having problems with my API. I just did a download from the My Characters tab on the left hand side to get my wallet transactions and only two showed up. They are both from June 26, so almost one month ago, but I wonder why there are only two? Where are my other 998 transactions? I also downloaded the journal, and it had 548 returns, including market transaction from April 27th. I looked at the Journal Walking site but that is all greek to me. I don't quite understand exactly what I am supposed to do. Do I have to write some sort of script or program that goes back into CCP's database, or do I have to create a webpage or some other process using XML, which again I am lost on?

Sugar Jugs	Posted - 2008.07.23 02:49:00 - [11] If your characters transactions are corp based, they will NOT show in the personal transactions export. You need to have director access to export your corp transactions history afaik, otherwise it may be possible without director but with the correct roles for the wallet you use.
Ambo Dirty Deeds Corp. Axiom Empire	Posted - 2008.07.23 07:47:00 - [12] It sounds like that's all the transactions and journal entries available to you. Journal/transaction walking is only useful if you get 1000 entries returned by your first query. The trouble is that CCP limits the amount of data available through the API. You CANNOT go back beyond a certain point with it. The documented rules for where this point falls are described above. However, there also seems to be times when there is a hard cutoff at a certain date. This is not due to changing to a new generation of journal IDs as the IDs have not changed. In fact we have no idea why it happens, just that it has happened. -------------------------------------- Trader? Investor? Just want to track your finances? Check out EMMA
Neidhardt Foster University of Caille	Posted - 2008.08.20 09:14:00 - [13] sorry to dig out this slightly dated thread, but I just stumbled across that problem and would be glad for some clarifications on it. I returned from my holiday trip an when updating my wallet transactions in EMMA i get 1000 Entries and then a "Warning no more entries found", my Journal however got updated with more than 1000 transactions until the "found key allready in database" message appeared. Strangely looking at my transactions in EMMA does not reveal an unusaly big gap in transactions, I have transactions for every day (since I was away on holiday only 4-5 transactions a day one week ago seems normal as well). At the cutoff point of 1000 transactions I have a delay of 4 hours between the transactions. The API page gives me the following error: "# error WeekBackAlready: Already returned one week of data: retry after 2008-08-20 09:15:05." If I understand the information supplied in this thread correct, Eve stores the market transactions for 1 week (no limit), if information is older than a week it stores the last 1000 entries. My question are: - Is any information not covered by the above mentioned rules, lost? - Or was there just another "database maintenance" from CCP? - Is there any service that automaticly backups my market logs (a player run eve-api mirror or something like that)? Because I really would like to have my transaction history complete, but it will happen that I'm not able to do a backup once a week. - Was I just lucky that I don't have a big gap in my transactions? Or is there something else I miss? - Am I correct in my assumption that API access is read only for everything? I tested a different tool, before I updated EMMA, but I don't think this could not be the reason for the 1000 transactions cutoff.
Chira Netanru	Posted - 2008.08.20 15:34:00 - [14] REFID (returned in the XML data) is not a primary key, as I've discovered after trying to report a 'bug' in the API. It is a foreign key. The REFID will be duplicated in a handful of instances, from things like buying off your own sell orders, selling to your own buys, or transfering isk from one corporate wallet division to another in the same corp. Basically think of REFID as a transaction ID where the XML you are viewing is more of ledger records. Another problem I've run into is getting '0' in my division column. I haven't looked at the raw XML yet to see if this is a problem caused by the API, or by Ray's [SEXY] wallet that I use. My alt corp generates about 60,000 journal entries a week, FWIW. as far as 'old' data disappearing, I bet you'll find some correlation to generating a new API key...
Shar Tegral	Posted - 2008.08.20 15:46:00 - [15] Originally by: Chira Netanru as far as 'old' data disappearing, I bet you'll find some correlation to generating a new API key... Nope... not this. Just used sexy on my alt-main (HawkBlade) for the very first time, in fact very first time using his full api key with anything, and only 20 items have returned. The API key has never been regenerated and I assure you, he's bought stuff constantly for years. Point of reference: The 20 items were things purchased in August. Nothing from before.
Chira Netanru	Posted - 2008.08.20 16:36:00 - [16] How far back does his ingame wallet go?
Shar Tegral	Posted - 2008.08.20 16:41:00 - [17] Originally by: Chira Netanru How far back does his ingame wallet go? Same distance.
Chira Netanru	Posted - 2008.08.20 18:18:00 - [18] So uhh.... Why you cryin'?
TornSoul BIG Libertas Fidelitas	Posted - 2008.08.20 18:25:00 - [19] Originally by: Shar Tegral Originally by: Chira Netanru How far back does his ingame wallet go? Same distance. Which kinda tells you the whole story. There simply isnt any older data in the DB. Ie. it's not API specific. (might be in some backup somewhere - but not in the active DB table) CCP cull their DB tables every now and again - Due to space and efficiency of the DB. This happens with mails as well - Not just wallet/journal stuff.
Matalino Gallente Ki Tech Industries	Posted - 2008.08.20 18:46:00 - [20] Here is what I know about journal/transaction history. Each record has an 32bit ID field. Every so often, CCP needs to run a database task to reset that field so that they don't run out of ID's. When they run that task, old data is archived. Data that is less than a week old is certainly kept available. Older data is not. I don't know exactly what the cut off is when they do the purge, but I suspect that it is actually closer to a coule months than a week. When you access the API key, you will always be able to access the past weeks transactions. If you have had less than 1000 transactions in the past week, you can access up to 1000 entries, or back until the previous data purge if you have less then 1000 transactions since then.

Shar Tegral	Posted - 2008.08.20 18:53:00 - [21] Originally by: Chira Netanru So uhh.... Why you cryin'? Awwwww, they're so cute when they are this young. But, I wasn't crying. I was simply reporting that believed standards that govern the API are not as they were once believed. Of course this point was brought up in an older thread elsewhere... if you had read this thread. What's the relevance to you? I don't know and, more likely, I don't care.
SencneS Amarr Rebellion Against big Irreversible Dinks	Posted - 2008.08.20 19:01:00 - [22] I don't know about anyone else, but I can access my journal up all the way back to 14th of June, 2007. I go about 40-50 transactions sales a day so the journal will have tax entry as well. Making it between 80-100 entries a day. Goto your wallet, goto your journal, Under the Date, make it read 2007 and press enter. Then press load. It will take you to last year.
TornSoul BIG Libertas Fidelitas	Posted - 2008.08.20 22:36:00 - [23] Originally by: Matalino Each record has an 32bit ID field. Every so often, CCP needs to run a database task to reset that field so that they don't run out of ID's. I'm taking a guess here but: I'm fairly sure they cull wallet and other tables before running into the 32bit problem. As that's really a whole other kettle of fish. Even when they do the "regular" culling of wallet (and other) data, the refID's for wallet entries do not get reset each time as well. But it's when the refID (and/or other ID's) run into the 32bit barrier they need to do the "refresh" to free up ID's. And at the same time they are probably doing the regular house cleaning as well (might as well while at it).
Neidhardt Foster Sternenschauer AG	Posted - 2008.08.20 22:43:00 - [24] Originally by: SencneS I don't know about anyone else, but I can access my journal up all the way back to 14th of June, 2007. I go about 40-50 transactions sales a day so the journal will have tax entry as well. Making it between 80-100 entries a day. Goto your wallet, goto your journal, Under the Date, make it read 2007 and press enter. Then press load. It will take you to last year. As I wrote my journal data got not cut off, so probably journal data is treated different than wallet data, possibly it is persistent. But since I can't make the connection from the journal entry to the sell/buy order this is of no use for the purpose of having a complete history of my business, which I would like to have.
Pink Kiwi	Posted - 2008.08.21 03:58:00 - [25] From what I can tell, wallet transactions have 3 limits in place. First is one week, then up to 1000 entries, then up to 1 month maximum of data. Anything older then 1 month is not available from the API regardless of number of entries. It has nothing to do with ID numbers. It consistently cuts off at 1 month. I don't have enough old Wallet Journal downloads to analyze, but at a quick glance it seems to have a maximum of up to 3 months.

Pages: [1] :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.