|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 01:59:00 -
[1]
Hi,
I've been sitting on this for quite while, but only recently CCP gave the green light for releasing what has been my pet project for many years.
So, I hereby proudly introduce....
RΞVΞRENCE REVERse ENgineered Cache Explorer
This is a strictly x86/x64 Python 2.6 library (Python 3.x support maybe in the future), targeted mainly at developers using Windows, although it has been tested successfully on Linux.
FEATURES
- High-performance iterative cache/bulkdata decoder.
- 100% compatibility with all bulkdata, cache and settings files.
- Programmatic access to data tables.
- Transparent loading of bulkdata on accessing tables.
- Simultaneous handling of data from multiple EVE installations/versions.
- Container classes for all data items found in cache and bulkdata.
- Offline RemoteSvc calls, provided the relevant cachefiles exist. Note that this software DOES NOT interact with the EVE Online client or server.
- EmbedFS (.stuff) file support.
- Various EVE related utility functions and constants.
The library provides programmers with an interface very similar to what EVE uses itself under the hood (yes, CCP reviewed the code and actually didn't mind it), with some variations where I deemed necessary for usability, efficiency, etc, as EVE was never designed to operate on the dataset of multiple EVE installations at once.
It's licensed under the terms of the BSD license so you can do almost anything you want with it (if you're mad enough to port it to other languages, let me know!)
DOWNLOAD The project is available on github here: http://github.com/ntt/reverence
There's a windows installer and source distribution in the downloads section.
USAGE Some examples have been provided (see link above), and some instructions are in the README.txt in the distribution. You'll have to excuse me for the lack of depth of the documentation, but this has been a fairly rapidly changing project over its lifetime (it 'settled' only recently). I'll probably write more thorough documentation SoonÖ.
I hope you'll enjoy this thing as much as I did creating it.
THANKS Special thanks go to CCP for granting permission to release this product, even though it is borrowing heavily from EVE Online's design.
_
Got Item? | EVE API? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 04:06:00 -
[2]
Note that there currently appears to be an issue when attempting to load the entire CachedMethodCalls folder (may cause python to crash), so there's something in there that doesn't play nice. I'll get on that later. For now just don't try to load that particular folder.
This isn't something normally done anyway, so it will not affect most people at all :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 13:36:00 -
[3]
Edited by: Entity on 03/10/2009 13:41:14
Originally by: Chyzwar Amarr Hmmm very interesting stuff here but:
error: Unable to find vcvarsall.bat
I suppose it was because of the absence of visual studio compiler, right?
Yeah, use the installer, or download MS VC++ 9.0 Express (it's free anyway)
Originally by: Kaladr (I'll have to borrow some features of course ;))
On a side note: Reverence is a play on Reference (intentional) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.05 22:47:00 -
[4]
Originally by: Vessper Anyway, the calculation of the cache location in Windows is incorrect when using folders containing a space (e.g. Program Files). This seems to be resolved by altering the cache.py file to state:
if os.name == "nt": cacheFolderName = root.lower().replace(":", "").replace("\\", "_").replace(" ","_")
Fixed, thanks _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.09 01:46:00 -
[5]
Originally by: Matthias Dete Hmm... kinda got it working (was problems with the cache not being in the right place in Windows 7)
When iterating over the invTypes however, now and then i get:
UnicodeEncodeError: `charmap` codec can't encode character u'\u2013' in position 8: character maps to <undefined>
Any suggestions...
Uh, names and descriptions can contain unicode characters. if your means of displaying cant handle it (such as cmd.exe console), you get that sort of error.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.26 02:07:00 -
[6]
version 1.0.1 is now available (see OP). This version fixes the aforementioned crash bug.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.28 04:10:00 -
[7]
Originally by: Tonto Auri I'd like to ask simple question... could it allow me to write settings files back so EVE could read them?
Nope. This library doesn't and won't include write capability, as that's a bit dodgy (EULA forbids modifying game files). I would recommend simply backing up your settings files instead :)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.12.19 15:11:00 -
[8]
Oddly enough nobody seems to have issues with this toolkit?!
How about feature suggestions then? :P
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.04.02 12:42:00 -
[9]
Anti-thread-expire bump :)
I may have to release an update soonÖ with the planetary interaction stuff coming in...
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.04.02 13:56:00 -
[10]
Originally by: Catari Taga
Originally by: Entity Anti-thread-expire bump :) I may have to release an update soonÖ with the planetary interaction stuff coming in...
Since you bumped it, I always meant to post an issue I had with making it work on non-TQ installs because at least on my system the cache folder is different than what your program was expecting. To make it work I replaced line 94 in cache.py with the following:
if servername == "tranquility": cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName, "cache") else: cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName + "_" + serverip, "cache")
I would guess that's because there's 2 ways to fire up sisi and they both produce a different folder (by name and by ip). I'll have to think of something clever to get around that :P _
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.05.10 20:46:00 -
[11]
Originally by: Miss AmarrPriceCheck
Originally by: Ix Forres
Originally by: Miss AmarrPriceCheck This library is excellent and I look forward to its Tyrannis update. Thanks, Entity!
Shouldn't need updating for Tyrannis, should work out of the box.
Oh, I was just going by what Entity said, above: "I may have to release an update soonÖ with the planetary interaction stuff coming in..."
If it continues to work fine on patch day, all the better.
It'll work. That was mostly referring to any possible new tables relating to PI, so I might have to add some table definitions. They can be manually accessed though, it's just less convenient.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.05.15 13:58:00 -
[12]
Edited by: Entity on 15/05/2010 13:59:03
Originally by: cinderbrood Edited by: cinderbrood on 15/05/2010 04:38:11 Will we ever see a pure python version of this without the c/c++ extensions?
(I want to run this on IronPython) but alas they lack .pyd support.
Ie will we ever see a pure python version of your blue.pyd?
would open up cache support to the .net world as well. (albeit in a hacky manner)
Heh. It actually used to be pure Python, it was mostly a pet research project. Worked fine, and I've used it like that for a long time, but there's a couple of problems with it:
- I'd end up having to maintain the same thing in two different languages.
- Memory usage goes through the roof because of Python data structure overhead.
- Without acceleration (Psyco jit compiler) it is painfully slow.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.13 19:19:00 -
[13]
Originally by: Jita Grrl
Originally by: Entity Edited by: Entity on 03/10/2009 02:09:45 100% compatibility with all bulkdata, cache and settings files.
What exactly do you mean when you said settings files?
I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.
Is this something Reverence can do or am I barking up the wrong tree?
Yeah it can do that, simply do something like
obj = blue.marshal.Load(contents_of_settings_file)
settings files are mostly just dictionary mappings.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.17 13:35:00 -
[14]
Edited by: Entity on 17/06/2010 13:35:44 A windows installer and source distribution for version 1.1.0 are now available (see OP for location) for your convenience.
Version 1.1.0 fixes various bugs and other issues and is compatible with Tyrannis, and adds a couple of features. The most useful change being that DBRows now have a string representation, which is handy for debugging and checking out table contents.
Check the commit history on github for more details.
Note: Users already using the current version directly from the repository don't have to do anything.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.17 15:47:00 -
[15]
Originally by: Veevi Fenikusu Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)
I would just monitor the folder for new files and only check those after examining the existing ones. Also, you can decode the files manually with blue.marshal.Load(data) instead of using cachemgr. A further optimization would be to just inspect the binary data and see whether it's a market order before loading/decoding it. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.20 12:06:00 -
[16]
Originally by: Gogo cyno I was playing around with this and had a question about the cachemgr's GetCacheFileName method:
Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?
GetCacheFileName is kind of for internal usage and manual loading (by loading the file and decoding with blue.marshal.Load). If you know the cache key of a file (the tuple you mentioned), you can do this if the file is in CachedMethodCalls:
obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )
and this if it's in CachedObjects:
obj = cache.LoadCachedObject( ('marketProxy', 'GetCorporationOrders', 1234) )
Note that LoadCacheFolder is a one-shot thing and it actually loads all the data at once, so it is very inefficient if you only use one object. The method is provided mostly for debugging/dumping.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 18:52:00 -
[17]
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 20:04:00 -
[18]
Originally by: Entity
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
Hm, it looks like it's actually a decoding problem, but it's so incredibly subtle I don't understand why it's failing.
The problem: cPickle produces a different pickle string for two seemingly identical tuples.
Example: cachedmethodcall for t2 expanders in Forge:
('marketProxy', 'GetOrders', 10000002, 1319)
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
The only difference here are the objectIDs differing. Even comparing the key to the constant ('marketProxy', 'GetOrders', 10000002, 1319) produces True.
This shouldn't really matter, but apparently it does.
A workaround is to re-tuple the key by doing key = key[:] before using it.
At the moment I have no idea why it behaves like that, and I'm tempted to use that workaround in the library.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 00:56:00 -
[19]
Originally by: Gogo cyno
Originally by: Entity
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
>> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) 'd58e.cache'
And yet another result. I tried using the workaround you mentioned with no success.
I get the distinct feeling cPickle has some undocumented behaviour.
what does cPickle.dumps(('marketProxy', 'GetOrders', 10000002, 1319)) give? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 01:35:00 -
[20]
oh wow. Found the cause of the problem. cPickle is not stable for given input depending on refcounts.
Originally by: picklebug.py
import cPickle
reffed = "xKITTENSx"[1:-1] print repr(cPickle.dumps(reffed))
print repr(cPickle.dumps("xKITTENSx"[1:-1]))
Produces:
F:\>picklebug.py "S'KITTENS'\np1\n." "S'KITTENS'\n."
Note the extra "p1\n" in the 1st case.
Very strange... Not sure how to solve this. seems to be a cPickle issue.
More worryingly, since the method I use to generate these cache filenames is exactly identical to what EVE does, I wonder whether EVE would have the same problem in some cases.
_
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 15:26:00 -
[21]
Yep, but not identical to cPickle. Consistency doesn't help if the string is still different, as the hash code will be different too. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:09:00 -
[22]
Update
I've found evidence of EVE also being affected by this, as I have found cache entries for calls to the same method, with different arguments, that exhibit the same pickling inconsistency:
"(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVX-Factor Ind\nI0\ntp3\n." <- only three PUTs "(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVNTT\np3\nI0\ntp4\n." <- four PUTs
So, it's pretty much impossible to know whether EVE held more than 1 reference to a string when it hashed the cache key, and as such there's no good way to solve this problem.
Recursively collecting all elements of the key into a new list so that their reference count is going to be at least 2 before using GetCacheFileName() does seem to fix it for most of the cache entries, but there's still going to be a handful that fail to produce the correct hash.
Try the following workaround, but note that you should only do this for the CachedMethodCalls folder.
def collect(key): foo = [] _collect((key,), foo.append) return foo
def _collect(items, append): for element in items: if type(element) in (list, tuple): _collect(element, append) append(element)
foo = collect(key) filename = cm.GetCacheFileName(key)
Guess I'll be poking some people in CCP about this.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:55:00 -
[23]
Originally by: Catari Taga
Originally by: Entity Guess I'll be poking some people in CCP about this.
Please follow the normal procedure and explain the bug to a bughunter first.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 22:21:00 -
[24]
yep, that works too.
I've send off some mails to the right people so hopefully the bug is going to be fixed sometime SoonÖ and i'll update Reverence :)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.18 12:32:00 -
[25]
Originally by: Induc Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?
Working on it. It compiles for 2.7 it seems. Will have to check EVE first once it's up, to see if the marshal format and cache key hashing are still the same.
Update SoonÖ _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.21 13:08:00 -
[26]
Hm well I couldn't find any problems with the 2.7 build.
git updated, I'll put a 2.7 binary distribution SoonÖ _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.09.04 11:43:00 -
[27]
2.6 and 2.7 binaries now available.
Fixed some stuff as well: - Reverence now attempts to import modules from its own package folder first before trying a global import to avoid conflicts with other packages. - Automatic cache path discovery now slightly more intelligent and won't be fooled by an empty cache folder in client installs when there's one in %LOCALAPPDATA%.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.13 17:18:00 -
[28]
Originally by: Xendrais Very nice library
Since the last patch 1.1, I get the following error:
Traceback (most recent call last): File "<stdin>", line 4, in <module> File "C:\Python27\lib\site-packages\reverence\config.py", line 376, in __get__ value = self.method(obj) File "C:\Python27\lib\site-packages\reverence\config.py", line 388, in method return self._loadbulkdata(tableName=(tableName or attrName), storageClass=storageClass, rowClass=rowClass, primaryKey=primaryKey) File "C:\Python27\lib\site-packages\reverence\config.py", line 740, in _loadbulkdata obj = self.cache.LoadObject(fullTableName) File "C:\Python27\lib\site-packages\reverence\cache.py", line 285, in LoadObject raise RuntimeError("cache file not found: %s" % fileName) RuntimeError: cache file not found: ce99.cache
This file "ce99.cache" exists in CachedObjects but not in CachedMethodCalls.
Any idea why and suggestions to correct?
Thanks
Reverence might be confused about the location of the cache (can happen if you use /LUA:OFF). Also, it seems that one of the patches cleaned out the client's BulkData folder. Which means that if you have not logged in properly at least once, reverence would give that sort of error. So, does it still fail? As my copy seems to work just fine.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.14 18:17:00 -
[29]
Originally by: Xendrais 2) Also under MachoNet, there are now two groups of folders: 87.237.38.200\247 and 127.0.0.1\249
The later group is the most current one. Reverence does not find this group but only the first one! Thus giving old information...
Any more idea?
127.0.0.1? what? are you running EVE through a local proxy or something? You'll have to use eve=blue.EVE(server="127.0.0.1") to access the data from there then.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.16 01:40:00 -
[30]
Originally by: Xendrais
Originally by: Induc
You need to provide the path to your eve installation too. >>> eve = blue.EVE("C:/...", server="127.0.0.1")
Now I get this and the path to EVE is good:
>>> eve = blue.EVE("C:\Program Files\CCP\EVE", server="127.0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python27\lib\site-packages\reverence\blue.py", line 130, in __init__ self.cache = cache.CacheMgr(self.root, self.server, machoVersion, cachepath)
File "C:\Python27\lib\site-packages\reverence\cache.py", line 218, in __init__ raise RuntimeError("Could not determine EVE cache folder location.") RuntimeError: Could not determine EVE cache folder location. >>>
Hrm, well, I think you're better off trying to figure out why you are getting a 127.0.0.1 folder in your cache instead of the tranquility IP. That's the real problem here. If you know how to reproduce this situation, let me know. I'm not getting a 127.0.0.1, so I don't know what is going on, and Reverence can't magically figure out where your files are in non-standard situations like yours.
The only thing I could think of as a workaround is providing cachepath=r"C:\path\to\eve's\cache\" to blue.EVE() as well. If you're not using LUA and are on XP, it would be r"C:\Documents and Settings\USERNAME\Local Settings\Application Data\CCP\EVE\mangled_copy_of_install_path\cache". Forgot where it is on Win7 but i'm sure you can find it :P
(On a side note, it's a good idea not to use backslashes bare in python strings. Escape them, use the r string prefix, or use forward slashes to avoid problems)
_
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.24 11:28:00 -
[31]
Originally by: D Spectre blah
Well yeah modifying the example to use CMC isn't going to work because CachedMethodCalls are different objects.
try something like this:
from reverence import blue
eve = blue.EVE("C:/path/to/eve") cm = eve.getcachemgr() crap = cm.LoadCacheFolder("CachedMethodCalls")
for key, obj in crap.iteritems(): if "GetOrders" in key: orders = obj['lret'] for order in orders[0]: # 0 for sell, 1 for buy orders print order.orderID, order.price, int(order.volRemaining)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.27 14:43:00 -
[32]
Originally by: Tanmoe I'm unable to find the documentation for the fields in the orders records. Most are obvious but i'm missing a few things.
for example : jumps don't appear to be right for buy orders. What is the issued value?
I believe it is the server's idea of distance. The client calculates jump distances with your current autopilot settings (which are entirely client-side) before displaying them in the market view anyway.
You should probably ignore that field and calculate it yourself if you require that info.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.19 18:53:00 -
[33]
Originally by: Blazing Fire From the readme it is not clear to me if this tool read only the static data or it can be used to read the market orders and market history. Can someone confirm.
I am a newbie .Net codder and I want to read market data in my app. and I have no idea if this Pythin library can be used on .Net app and how.
Is there a place where I can read about the actual structure of the EVE .cache files? Again I am interested in the market data.
It can read anything in the cache. which includes market orders/history as that stuff appears in the cache when you look it up ingame.
Reverence is not compatible with IronPython afaik (so much for .NET)
No documentation for specific cache objects exists.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.19 18:58:00 -
[34]
Notice to Reverence users:
It seems that with this patch, one or more of the static data tables died.
config.BulkData.ramtypematerials is one of the casualties, as the bill of materials seems to be created serverside by request now, judging by the new GetMaterialsForTypeWithActivity method I see in my cache.
This means cfg.prime() will fail (funnily enough on the very last table it loads), but apps using transparent loading that do not use the ramtypematerials table will continue to function as normal.
I'll update the library to not load ramtypematerials in cfg.prime() soonÖ
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.20 18:35:00 -
[35]
Originally by: Two step I'm pretty sure it just got renamed to invtypematerials or something similar. I was looking at diffs between tq (pre-patch) and sisi with the final patch and that is what I saw.
Ah, I must've read invtypereactions, I'm blind. Thanks :P A table name change still requires a reverence update though :-/
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.30 07:56:00 -
[36]
CCP's moving the handling of certain types of data structures from Python into C++ for performance. So, CFilterRowset is the C++ implementation of the python FilterRowset, and I have yet to write a class for it. I haven't looked at the structure yet, will do SoonÖ
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.03 05:49:00 -
[37]
If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.04 07:43:00 -
[38]
Originally by: Schry Evolon
Originally by: Entity If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.
Assuming you are "ntt" of the GitHub, I have sent it to your email addy.
Hmm that one works here. I guess it's one of the smaller changes I haven't put into a release yet. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.07 09:19:00 -
[39]
I've committed a potential fix for the CFilterRowset problem. Update and build from source if you want to test it.
Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.
Guess you'll find out soon enough ;) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.10 12:00:00 -
[40]
Originally by: Thart Edited by: Thart on 10/03/2011 10:37:25 Users of my program reports problem with cache reader based on Reverence.
Quote: [error] Updating market orders (cache)... [error] Error calling cache import tool. Code: 255
Quote: Traceback (most recent call last): File "evemarketordersfromcache.py", line 49, in <module> AttributeError: 'module' object has no attribute 'StackSize'
The changes to the inventory system in latest patch caused this. looking into it. _
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.10 20:57:00 -
[41]
I've pushed a fix for the inventory system changes. Update/rebuild your copy of Reverence from source to try it out. (No new installer of yet).
Thanks to Thart for providing test files :)
For the technical: The fix was not trivial; CCP gave the DBRow subsystem a new feature that basically implements Python properties.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.19 13:02:00 -
[42]
Originally by: Theodor Bliesheimer Any estimate when an updated installer will be released?
Reason I'm not doing this as frequently as I should (read: every build) is because it's a pain to manually upload files to github. I'll upload new installers when I've finished figuring out their upload mechanics so I can script it :P
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.21 11:47:00 -
[43]
Originally by: Zeta Zhul A couple things.
1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.
Is there actually a reason why it deletes every single file regardless of the extension or type?
Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.
2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.
sigh.
3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.
Wrong thread? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.18 13:44:00 -
[44]
Originally by: Alaya Carrier Edited by: Alaya Carrier on 18/04/2011 10:00:23 Is this project dead like libevecache seems to be?
It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source.
It's not dead. Most of my projects simply require very little to no maintenance, even across patches. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.19 21:00:00 -
[45]
Edited by: Entity on 19/04/2011 21:00:45
Originally by: Alaya Carrier
Traceback (most recent call last): File "C:\Python26\EvEDump.py", line 54, in <module> for history_item in blah['lret']: TypeError: string indices must be integers, not str
I uninstalled the library. I downloaded the git current release, rebuilt, installed with no errors and...
... now I still get the same associative array error.
What gives?
I'd wager a guess that the structure of that object changed when they patched. I do not control how CCP represents data in cache. This is not a bug.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.25 10:35:00 -
[46]
Edited by: Entity on 25/04/2011 10:35:35
Originally by: Sverre Haakonson Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.
Originally by: Entity
DOWNLOAD The project is available on github here: http://github.com/ntt/reverence
There's a windows installer and source distribution in the downloads section.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 02:09:00 -
[47]
Originally by: Mike'P Hi guys,
(although this will probably only make sense to Entity!)
Interesting. Can you put this file somewhere I can download it? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 11:07:00 -
[48]
Originally by: Risingson i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
I mean the cache file causing the crash, silly... _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 11:48:00 -
[49]
Originally by: Risingson
Originally by: Entity
Originally by: Risingson i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
I mean the cache file causing the crash, silly...
uhmm, confused by your answer... is it somehow possible to get the data out of those 2 cache files?
Thanks for reminding me why I do not have a customer support job
Originally by: Risingson "narrowed it down to one cache file (d03c.cache - it has the string 'charGetNewTransactions' in the header)"
I want that -file-. Put it somewhere so I can inspect it myself.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 12:18:00 -
[50]
Edited by: Entity on 17/05/2011 12:23:48 Edited by: Entity on 17/05/2011 12:19:31
Originally by: Risingson Edited by: Risingson on 17/05/2011 12:09:05
Originally by: Entity Thanks for reminding me why I do not have a customer support job
hehe. sorry should have thought of that by myself.... 86b8.cache and 9554.cache, datadump.py
thank you!
Ah, yeah, they're not database tables, so datadump.py does not dump those. You should write code yourself to deal with these.
do
obj = blue.marshal.Load(file)
then obj[1]['lret'] is a list of tuples it seems. other than the systemID and name I don't know what the 2 other values in there are though.
Also, seems I got confused by your earlier post, sorry for that ^.^ _
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.18 00:54:00 -
[51]
Originally by: Mike'P
Originally by: Entity
Interesting. Can you put this file somewhere I can download it?
Well, I can't give you that particular file as it has now changed and doesn't exhibit the problem. However, I can give you another file which is even more screwy - it reports 4 shared objects, but if you get the old hex dumper out, there is quite clearly not 4 lowball int32_t's at the end of the file.
Grab it here....
http://www.team59.net/66d6.cache
hm, looking at that file, it shows 2 ints at the end, and the count (the int after the very first byte) also says 2, and it decodes without error here (XP 32bit) I'm confused now :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.18 17:17:00 -
[52]
Originally by: Mike'P Anyway, if you try the link again :-
Ok, I've checked that file and did some occult experiments. I can safely say it's either corruption, a failed save, or a bug in EVE. See, the map counter requires that many ints at the end of the file, because there will be that many distinct TYPE_REF tokens in the stream. I think not even EVE itself will know how to decode files with such corrupted maps; there is strong evidence for this in blue.dll in the form of a string saying "Bogus map data in marshal stream".
There is no fix for this, but I will make it detect this case and throw a proper exception. (probably one saying "Bogus map data in marshal stream" )
I'll update Reverence SoonÖ.
Thank you for pointing out this issue :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.25 17:04:00 -
[53]
1.2.2 released. The only change is the fix for the corrupt shared object map issue above, but updating doesn't hurt :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.23 22:16:00 -
[54]
Originally by: Sullinae busted for Incarna, needs to be updated for the new cache formats please update this great tool, I want to start making piles of isk with my trade finding script again!
It'd be nice if you could be a tad more specific? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.25 12:52:00 -
[55]
Originally by: Kandreath
File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal l raise ValueError(str(list(result.keys()))) ValueError: [u'path']
Any Idea?
Hmmm, I looked around a bit, and only found: http://stackoverflow.com/questions/4676728/value-error-trying-to-install-python-for-windows-extensions
Quote: If you have a 64 bit Python installation: Install "Microsoft Visual Studio 2008 Professional Edition" with the "X64 Compiler and Tools" option enabled.
I cannot test this myself yet as i'm using 32bit windows and too lazy to update. (I did buy a new laptop this week, however it's DOA, so so much for that idea)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.25 12:57:00 -
[56]
Originally by: Sullinae Getting an error: AttributeError: 'blue.DBRow' object has no attribute 'issued'
Seems to work other than that now.
Keep in mind CCP can and does change the structure and contents of cache files whenever they feel like. Errors like these are nearly always indicative of such a change.
In this case, CCP simply renamed the "issued" column to "issueDate". Quite far from 'busted for incarna' ;)
Originally by: Sullinae Thanks for all your hard work btw, this library rocks my socks./quote]
Cheers :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.07.27 01:35:00 -
[57]
I've just committed a few crash bug fixes. The fixes are for a reference count bug in virtual column names, and directly nested arbitrary-size containers being prematurely cut because of not clearing the TYPE_MARK state.
These bugs don't affect most (if any at all) users, so no new installers yet.
Note that Reverence is currently incompatible with the build on Singularity due to CCP deploying a shiny new bulkdata management system. The necessary changes have been done to make Reverence work with that, but I will not push these changes until TQ is updated, as things may still change.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.07.29 13:16:00 -
[58]
Originally by: Fena Echerie Hi,
I just downloaded your lib and tried datadump.py. I see lots of xml files (mostly bulk data) but the market data I want is missing. Snip of market data related errors: MethodCall.proxy.marketProxy.GetRegionBest.10000016L ... UNSUPPORTED (dbutil.RowDict) MethodCall.proxy.marketProxy.GetMarketGroups ... FAILED: Unsupported type // I edited datadump.py to report error type
When looking for a solution I found your last post, saying the windows installers are out of date, so I tried to compile the sources myself. However: > python setup.py build <snip> building 'reverence._blue' extension error: Unable to find vcvarsall.bat
Since I have Visual Studio 10 Professional installed and already tried adding the location of vcvarsall.bat to the PATH variable, I'm kind of lost now. Any help is appreciated :)
btw: Great work! Reverence makes a much better and cleaner impression than libevecache, which is passing/returning std::vectors by value all the time...
The dump script was made for bulkdata, not for dynamic stuff like market data. If you want to process market data you will have to write your own code for it (there's several examples in this thread iirc)
Also, to compile it you need VS9 I think. Python's build scripts are picky :)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.08.24 22:15:00 -
[59]
Originally by: Uberfrau
Originally by: Manathera Edited by: Manathera on 18/12/2010 21:13:13 I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable. MarketReader Please let me know of any errors as this is the first python code I've ever written.
I get a Segmentation fault: 11 :]
Quite possibly an issue that's been resolved already. If you can, checkout the repository and build from that. otherwise just wait for the new release (which should be when CCP switches over to the new bulkdata system) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.01 22:04:00 -
[60]
Reverence 1.3.0 now available (including installers). This release is compatible with the new Incarna 1.1 bulkdata system (but is backwards compatible).
_
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 15:46:00 -
[61]
yeah evelocations and eveowners are broken because I messed up with the DBRow initialization <_<. Fix SoonÖ _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 17:37:00 -
[62]
Edited by: Entity on 02/09/2011 17:37:13 Version 1.3.1 now available.
This hopefully fixes the eveowners and evelocations tables (I tested it though, so it should work fine ;)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 18:38:00 -
[63]
Originally by: Aineko Macx Have you ever considered rewriting blue in python to remove the dependency on a compiled C extension?
Reverence actually used to be pure python before I released it. It works, but it's terribly slow and uses a huge amount of memory to load tables.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.03 20:31:00 -
[64]
Originally by: Jaff Rockline
So - my question is twofold. 1. What must I do to fix this issue? 2. Is there a way I can list the current object attributes myself from the files so I can make the needed edits on my own?
Thanks for the help - and again - great work!
Indeed, attribute errors are pretty much always changes by CCP. To see the attributes on a row, just print the row itself to console. It should give some output like this:
DBRow(someAttribute:1,someOtherAttribute:2,...bladiblabla)
and you should be able to see what to do from there.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.03 21:15:00 -
[65]
Edited by: Entity on 03/09/2011 21:26:47
Hm, ah, I see what the issue is. It's due to the way those tables are made now. Not entirely sure what to do about this. I'll look into it.
Right. It's actually a tiny oversight on my part. See, I use a slightly different method for loading tables than EVE does, which is ever so slightly faster and adds more features and flexibility to the datasets, but for a few of those recently changed tables I used the normal (compatible) way, which is more strict about things (and possibly bugged, but nobody should be using compatibility mode anyway).
Fix SoonÖ _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.04 15:59:00 -
[66]
1.3.2 available.
lets you use the "name" attribute alias for eveowners and evelocations again :) _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.08 18:18:00 -
[67]
New forums thread: https://forums.eveonline.com/default.aspx?g=posts&t=6501 _
Got Item? | EVE API? | Cache? |
|
|
|