Pages: 1 2 3 4 5 6 7 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 01:59:00 -
[1]
Hi,
I've been sitting on this for quite while, but only recently CCP gave the green light for releasing what has been my pet project for many years.
So, I hereby proudly introduce....
RΞVΞRENCE REVERse ENgineered Cache Explorer
This is a strictly x86/x64 Python 2.6 library (Python 3.x support maybe in the future), targeted mainly at developers using Windows, although it has been tested successfully on Linux.
FEATURES
- High-performance iterative cache/bulkdata decoder.
- 100% compatibility with all bulkdata, cache and settings files.
- Programmatic access to data tables.
- Transparent loading of bulkdata on accessing tables.
- Simultaneous handling of data from multiple EVE installations/versions.
- Container classes for all data items found in cache and bulkdata.
- Offline RemoteSvc calls, provided the relevant cachefiles exist. Note that this software DOES NOT interact with the EVE Online client or server.
- EmbedFS (.stuff) file support.
- Various EVE related utility functions and constants.
The library provides programmers with an interface very similar to what EVE uses itself under the hood (yes, CCP reviewed the code and actually didn't mind it), with some variations where I deemed necessary for usability, efficiency, etc, as EVE was never designed to operate on the dataset of multiple EVE installations at once.
It's licensed under the terms of the BSD license so you can do almost anything you want with it (if you're mad enough to port it to other languages, let me know!)
DOWNLOAD The project is available on github here: http://github.com/ntt/reverence
There's a windows installer and source distribution in the downloads section.
USAGE Some examples have been provided (see link above), and some instructions are in the README.txt in the distribution. You'll have to excuse me for the lack of depth of the documentation, but this has been a fairly rapidly changing project over its lifetime (it 'settled' only recently). I'll probably write more thorough documentation SoonÖ.
I hope you'll enjoy this thing as much as I did creating it.
THANKS Special thanks go to CCP for granting permission to release this product, even though it is borrowing heavily from EVE Online's design.
_
Got Item? | EVE API? |
Arkady Sadik
Minmatar Gradient Electus Matari
|
Posted - 2009.10.03 02:37:00 -
[2]
Thank you very much, Entity. This is awesome.
Also, thank you very much, CCP, for allowing this to be published. Not many companies would do that.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 04:06:00 -
[3]
Note that there currently appears to be an issue when attempting to load the entire CachedMethodCalls folder (may cause python to crash), so there's something in there that doesn't play nice. I'll get on that later. For now just don't try to load that particular folder.
This isn't something normally done anyway, so it will not affect most people at all :) _
Got Item? | EVE API? | Cache? |
Kaladr
Amarr
|
Posted - 2009.10.03 05:56:00 -
[4]
Thanks a lot for this Entity. May Reverence and libevecache pave the way toward a more glorious future
(I'll have to borrow some features of course ;)) ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Chyzwar Amarr
|
Posted - 2009.10.03 10:51:00 -
[5]
Hmmm very interesting stuff here but:
running install running build running build_py running build_ext building 'reverence._blue' extension error: Unable to find vcvarsall.bat
I suppose it was because of the absence of visual studio compiler, right?
|
Robacz
Essence Enterprises
|
Posted - 2009.10.03 11:38:00 -
[6]
Thank you for releasing this, best MD post ever.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.03 13:36:00 -
[7]
Edited by: Entity on 03/10/2009 13:41:14
Originally by: Chyzwar Amarr Hmmm very interesting stuff here but:
error: Unable to find vcvarsall.bat
I suppose it was because of the absence of visual studio compiler, right?
Yeah, use the installer, or download MS VC++ 9.0 Express (it's free anyway)
Originally by: Kaladr (I'll have to borrow some features of course ;))
On a side note: Reverence is a play on Reference (intentional) _
Got Item? | EVE API? | Cache? |
Rilwar
BlackStar Industrial
|
Posted - 2009.10.03 13:56:00 -
[8]
This = The AwesomeSauce v2.0
It's part amazing, and part holy ****. \o/
If you didn't already own every ISK in EVE I'd send you some =x
-------------------------------------------------
Mitnal was here. |
Vessper
SI Radio
|
Posted - 2009.10.05 20:59:00 -
[9]
Nice work!
Just been playing around with it and found an issue but I couldn't find how to post an issue on the website without subscribing to some plan!
Anyway, the calculation of the cache location in Windows is incorrect when using folders containing a space (e.g. Program Files). This seems to be resolved by altering the cache.py file to state:
if os.name == "nt": cacheFolderName = root.lower().replace(":", "").replace("\\", "_").replace(" ","_")
- - - - - - EveHQ Character App |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.05 22:47:00 -
[10]
Originally by: Vessper Anyway, the calculation of the cache location in Windows is incorrect when using folders containing a space (e.g. Program Files). This seems to be resolved by altering the cache.py file to state:
if os.name == "nt": cacheFolderName = root.lower().replace(":", "").replace("\\", "_").replace(" ","_")
Fixed, thanks _
Got Item? | EVE API? | Cache? |
|
Matthias Dete
Oursulaert Industries
|
Posted - 2009.10.08 19:51:00 -
[11]
So... I've installed Python and Reverence...
Can someone explain to me where it's installed and how to get it to work?
Thanks
---------------------------------------------- Sig: Umbra Congregatio is now recruiting!!Check here
|
Matthias Dete
Oursulaert Industries
|
Posted - 2009.10.08 20:17:00 -
[12]
Hmm... kinda got it working (was problems with the cache not being in the right place in Windows 7)
When iterating over the invTypes however, now and then i get:
UnicodeEncodeError: `charmap` codec can't encode character u'\u2013' in position 8: character maps to <undefined>
Any suggestions... ---------------------------------------------- Sig: Umbra Congregatio is now recruiting!!Check here
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.09 01:46:00 -
[13]
Originally by: Matthias Dete Hmm... kinda got it working (was problems with the cache not being in the right place in Windows 7)
When iterating over the invTypes however, now and then i get:
UnicodeEncodeError: `charmap` codec can't encode character u'\u2013' in position 8: character maps to <undefined>
Any suggestions...
Uh, names and descriptions can contain unicode characters. if your means of displaying cant handle it (such as cmd.exe console), you get that sort of error.
_
Got Item? | EVE API? | Cache? |
LarcatOfRens
|
Posted - 2009.10.19 02:43:00 -
[14]
Is this useable for a non IT-Person to extract market history date?
High/Low/Volume/Average.
-Larcat
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.26 02:07:00 -
[15]
version 1.0.1 is now available (see OP). This version fixes the aforementioned crash bug.
_
Got Item? | EVE API? | Cache? |
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2009.10.26 22:47:00 -
[16]
I'd like to ask simple question... could it allow me to write settings files back so EVE could read them? I *really* hate the times when CCP throwing away my time spent on setting up interface, I want some way to keep it intact through CCP's carelessness. -- Thanks CCP for cu |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.10.28 04:10:00 -
[17]
Originally by: Tonto Auri I'd like to ask simple question... could it allow me to write settings files back so EVE could read them?
Nope. This library doesn't and won't include write capability, as that's a bit dodgy (EULA forbids modifying game files). I would recommend simply backing up your settings files instead :)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.12.19 15:11:00 -
[18]
Oddly enough nobody seems to have issues with this toolkit?!
How about feature suggestions then? :P
_
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Rough Necks
|
Posted - 2009.12.19 16:32:00 -
[19]
Edited by: Catari Taga on 19/12/2009 16:33:45
Originally by: Entity Oddly enough nobody seems to have issues with this toolkit?!
How about feature suggestions then? :P
It probably does not have many users. I tried it and it worked fine and indeed without issues, great job!
If you are looking to do more work on it how about a function to dump entire tables in datadump compatible sql/mysql format? That's probably what most people want from it anyway.
|
Cyaxares II
|
Posted - 2010.02.11 23:35:00 -
[20]
Extracting market history data with reverence Linkage.
|
|
Femaref
Armageddon Day
|
Posted - 2010.02.14 01:11:00 -
[21]
Originally by: Catari Taga Edited by: Catari Taga on 19/12/2009 16:33:45
Originally by: Entity Oddly enough nobody seems to have issues with this toolkit?!
How about feature suggestions then? :P
It probably does not have many users. I tried it and it worked fine and indeed without issues, great job!
If you are looking to do more work on it how about a function to dump entire tables in datadump compatible sql/mysql format? That's probably what most people want from it anyway.
If you'd check around the examples, there already is one doing exactly that.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.04.02 12:42:00 -
[22]
Anti-thread-expire bump :)
I may have to release an update soonÖ with the planetary interaction stuff coming in...
_
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Rough Necks
|
Posted - 2010.04.02 13:44:00 -
[23]
Edited by: Catari Taga on 02/04/2010 13:45:06 Hadn't seen this post:
Originally by: Femaref If you'd check around the examples, there already is one doing exactly that.
Not quite, I was proposing a datadump compatible format, which the example does not quite provide (you would basically need to add a static table to the tool with the current datadump format and the relationships and keep that updated after every change to the structure of either the dump or the game data). Also there is a bug with the example in that it does not add the ";" terminator after each insert.
Originally by: Entity Anti-thread-expire bump :) I may have to release an update soonÖ with the planetary interaction stuff coming in...
Since you bumped it, I always meant to post an issue I had with making it work on non-TQ installs because at least on my system the cache folder is different than what your program was expecting. To make it work I replaced line 94 in cache.py with the following:
if servername == "tranquility": cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName, "cache") else: cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName + "_" + serverip, "cache")
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.04.02 13:56:00 -
[24]
Originally by: Catari Taga
Originally by: Entity Anti-thread-expire bump :) I may have to release an update soonÖ with the planetary interaction stuff coming in...
Since you bumped it, I always meant to post an issue I had with making it work on non-TQ installs because at least on my system the cache folder is different than what your program was expecting. To make it work I replaced line 94 in cache.py with the following:
if servername == "tranquility": cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName, "cache") else: cachepath = os.path.join(path_buf.value, "CCP", "EVE", cacheFolderName + "_" + serverip, "cache")
I would guess that's because there's 2 ways to fire up sisi and they both produce a different folder (by name and by ip). I'll have to think of something clever to get around that :P _
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Rough Necks
|
Posted - 2010.04.02 14:21:00 -
[25]
Edited by: Catari Taga on 02/04/2010 14:25:00
Originally by: Entity I would guess that's because there's 2 ways to fire up sisi and they both produce a different folder (by name and by ip). I'll have to think of something clever to get around that :P
It's possible, in my case it's by IP and without /LUA:OFF. TQ is on E:\EVE. My two folders are thus:
%USERPROFILE%\Local Settings\Application Data\CCP\EVE\e_eve_tranquility %USERPROFILE%\Local Settings\Application Data\CCP\EVE\e_eve_singularity_87.237.38.50
Testing other combinations is unfortunately up to you - or just assume that whoever is using your library will probably be able solve the problem for their own system anyway. Thanks again for the library btw, it's been very useful.
|
Miss AmarrPriceCheck
|
Posted - 2010.05.08 17:59:00 -
[26]
This library is excellent and I look forward to its Tyrannis update. Thanks, Entity! |
Ix Forres
Caldari Vanguard Frontiers Intrepid Crossing
|
Posted - 2010.05.08 19:35:00 -
[27]
Originally by: Miss AmarrPriceCheck This library is excellent and I look forward to its Tyrannis update. Thanks, Entity!
Shouldn't need updating for Tyrannis, should work out of the box. -- Ix Forres EVE Application Developer EVE Metrics | accVIEW | I Tweet |
Miss AmarrPriceCheck
|
Posted - 2010.05.10 19:44:00 -
[28]
Originally by: Ix Forres
Originally by: Miss AmarrPriceCheck This library is excellent and I look forward to its Tyrannis update. Thanks, Entity!
Shouldn't need updating for Tyrannis, should work out of the box.
Oh, I was just going by what Entity said, above: "I may have to release an update soonÖ with the planetary interaction stuff coming in..."
If it continues to work fine on patch day, all the better. |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.05.10 20:46:00 -
[29]
Originally by: Miss AmarrPriceCheck
Originally by: Ix Forres
Originally by: Miss AmarrPriceCheck This library is excellent and I look forward to its Tyrannis update. Thanks, Entity!
Shouldn't need updating for Tyrannis, should work out of the box.
Oh, I was just going by what Entity said, above: "I may have to release an update soonÖ with the planetary interaction stuff coming in..."
If it continues to work fine on patch day, all the better.
It'll work. That was mostly referring to any possible new tables relating to PI, so I might have to add some table definitions. They can be manually accessed though, it's just less convenient.
_
Got Item? | EVE API? | Cache? |
cinderbrood
Caldari Delta Technologies And Development
|
Posted - 2010.05.15 04:37:00 -
[30]
Edited by: cinderbrood on 15/05/2010 04:38:11 Will we ever see a pure python version of this without the c/c++ extensions?
(I want to run this on IronPython) but alas they lack .pyd support.
Ie will we ever see a pure python version of your blue.pyd?
would open up cache support to the .net world as well. (albeit in a hacky manner)
Will Code For Isk. |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.05.15 13:58:00 -
[31]
Edited by: Entity on 15/05/2010 13:59:03
Originally by: cinderbrood Edited by: cinderbrood on 15/05/2010 04:38:11 Will we ever see a pure python version of this without the c/c++ extensions?
(I want to run this on IronPython) but alas they lack .pyd support.
Ie will we ever see a pure python version of your blue.pyd?
would open up cache support to the .net world as well. (albeit in a hacky manner)
Heh. It actually used to be pure Python, it was mostly a pet research project. Worked fine, and I've used it like that for a long time, but there's a couple of problems with it:
- I'd end up having to maintain the same thing in two different languages.
- Memory usage goes through the roof because of Python data structure overhead.
- Without acceleration (Psyco jit compiler) it is painfully slow.
_
Got Item? | EVE API? | Cache? |
cinderbrood
Caldari Delta Technologies And Development
|
Posted - 2010.05.15 20:40:00 -
[32]
Aww Damn it :P Speed wasnt an issue for me either ( Within reason i guess ) As for memory usage.. Yeah thats not going to get any better with IronPython i guess. Any chance of the old release being archived somewhere for those of us with pet projects themselfs that want to hack at it :P (Short of writing an entire .net version )
Writing my own cache reading libs was something im trying to avoid as its just an addon. not a required part of my little play toy.
Will Code For Isk. |
Miss AmarrPriceCheck
|
Posted - 2010.05.26 21:42:00 -
[33]
On a previously working script, I now have the following error post-Tyrannis:
Quote:
Traceback (most recent call last): File "C:\Documents and Settings\User\Desktop\scriptname.py", line 54, in <module> if str(cfg.evelocations.Get(key[2]).locationName) == "Lonetrek": File "C:\Python26\lib\site-packages\reverence\config.py", line 338, in __get__ value = self.method(obj) File "C:\Python26\lib\site-packages\reverence\config.py", line 606, in evelocations self._loadfrombulk("config.StaticLocations", rs, hint=1) File "C:\Python26\lib\site-packages\reverence\config.py", line 734, in _loadfrombulk rs.lines.extend(obj.lines) AttributeError: 'CRowset' object has no attribute 'lines'
Sorry the whitespace formatting in quoted error is obviously wrong, but you get the idea.
|
Kadesh Priestess
Scalding Chill
|
Posted - 2010.05.26 22:27:00 -
[34]
Entity updated reverence to work with tyrannis - you need to get and install latest version from git or if you prefer installer - wait for an update. Git version works perfectly for me.
|
Jita Grrl
|
Posted - 2010.06.13 12:06:00 -
[35]
Originally by: Entity Edited by: Entity on 03/10/2009 02:09:45 100% compatibility with all bulkdata, cache and settings files.
What exactly do you mean when you said settings files?
I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.
Is this something Reverence can do or am I barking up the wrong tree?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.13 19:19:00 -
[36]
Originally by: Jita Grrl
Originally by: Entity Edited by: Entity on 03/10/2009 02:09:45 100% compatibility with all bulkdata, cache and settings files.
What exactly do you mean when you said settings files?
I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.
Is this something Reverence can do or am I barking up the wrong tree?
Yeah it can do that, simply do something like
obj = blue.marshal.Load(contents_of_settings_file)
settings files are mostly just dictionary mappings.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.17 13:35:00 -
[37]
Edited by: Entity on 17/06/2010 13:35:44 A windows installer and source distribution for version 1.1.0 are now available (see OP for location) for your convenience.
Version 1.1.0 fixes various bugs and other issues and is compatible with Tyrannis, and adds a couple of features. The most useful change being that DBRows now have a string representation, which is handy for debugging and checking out table contents.
Check the commit history on github for more details.
Note: Users already using the current version directly from the repository don't have to do anything.
_
Got Item? | EVE API? | Cache? |
Veevi Fenikusu
Minmatar Wan Futuristics
|
Posted - 2010.07.09 11:38:00 -
[38]
This library is truly amazing, great work!
|
Veevi Fenikusu
Minmatar Fenikusu Trade
|
Posted - 2010.07.17 14:44:00 -
[39]
Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.17 15:47:00 -
[40]
Originally by: Veevi Fenikusu Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)
I would just monitor the folder for new files and only check those after examining the existing ones. Also, you can decode the files manually with blue.marshal.Load(data) instead of using cachemgr. A further optimization would be to just inspect the binary data and see whether it's a market order before loading/decoding it. _
Got Item? | EVE API? | Cache? |
|
Gogo cyno
|
Posted - 2010.07.19 16:45:00 -
[41]
I was playing around with this and had a question about the cachemgr's GetCacheFileName method:
Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.20 12:06:00 -
[42]
Originally by: Gogo cyno I was playing around with this and had a question about the cachemgr's GetCacheFileName method:
Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?
GetCacheFileName is kind of for internal usage and manual loading (by loading the file and decoding with blue.marshal.Load). If you know the cache key of a file (the tuple you mentioned), you can do this if the file is in CachedMethodCalls:
obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )
and this if it's in CachedObjects:
obj = cache.LoadCachedObject( ('marketProxy', 'GetCorporationOrders', 1234) )
Note that LoadCacheFolder is a one-shot thing and it actually loads all the data at once, so it is very inefficient if you only use one object. The method is provided mostly for debugging/dumping.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.07.20 15:29:00 -
[43]
Edited by: Gogo cyno on 20/07/2010 15:36:10
Originally by: Entity
obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )
Alas, I was hoping I was just doing it wrong. Whenever I try to load a specific method call in the above manner, I get a No Such File or Directory error and the filename will be different than what I know the cache file is. For example,
r = cachemgr.LoadCachedMethodCall( ('marketProxy', 'GetOrders', 10000002, 40) )
will produce that error. I know that method call is there because I can access it through loading the entire folder, but trying to load it specifically does not work.
cmc[('marketProxy', 'GetOrders', 10000002, 40)]
will return the correct information (with cmc = cachemgr.LoadCacheFolder("CachedMethodCalls") ) |
Saphira Khan
The Wretched.
|
Posted - 2010.07.30 16:44:00 -
[44]
I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them. www.eve-lotteries.com |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 18:52:00 -
[45]
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 20:04:00 -
[46]
Originally by: Entity
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
Hm, it looks like it's actually a decoding problem, but it's so incredibly subtle I don't understand why it's failing.
The problem: cPickle produces a different pickle string for two seemingly identical tuples.
Example: cachedmethodcall for t2 expanders in Forge:
('marketProxy', 'GetOrders', 10000002, 1319)
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
The only difference here are the objectIDs differing. Even comparing the key to the constant ('marketProxy', 'GetOrders', 10000002, 1319) produces True.
This shouldn't really matter, but apparently it does.
A workaround is to re-tuple the key by doing key = key[:] before using it.
At the moment I have no idea why it behaves like that, and I'm tempted to use that workaround in the library.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.08.02 20:38:00 -
[47]
Originally by: Entity
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
>> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) 'd58e.cache'
And yet another result. I tried using the workaround you mentioned with no success.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 00:56:00 -
[48]
Originally by: Gogo cyno
Originally by: Entity
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
>> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) 'd58e.cache'
And yet another result. I tried using the workaround you mentioned with no success.
I get the distinct feeling cPickle has some undocumented behaviour.
what does cPickle.dumps(('marketProxy', 'GetOrders', 10000002, 1319)) give? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 01:35:00 -
[49]
oh wow. Found the cause of the problem. cPickle is not stable for given input depending on refcounts.
Originally by: picklebug.py
import cPickle
reffed = "xKITTENSx"[1:-1] print repr(cPickle.dumps(reffed))
print repr(cPickle.dumps("xKITTENSx"[1:-1]))
Produces:
F:\>picklebug.py "S'KITTENS'\np1\n." "S'KITTENS'\n."
Note the extra "p1\n" in the 1st case.
Very strange... Not sure how to solve this. seems to be a cPickle issue.
More worryingly, since the method I use to generate these cache filenames is exactly identical to what EVE does, I wonder whether EVE would have the same problem in some cases.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.08.03 14:30:00 -
[50]
Using pickle seems to produce consistent results:
import pickle
reffed = "xKITTENSx"[1:-1] print repr(pickle.dumps(reffed))
print repr(pickle.dumps("xKITTENSx"[1:-1]))
produces:
"S'KITTENS'\np0\n." "S'KITTENS'\np0\n."
|
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 15:26:00 -
[51]
Yep, but not identical to cPickle. Consistency doesn't help if the string is still different, as the hash code will be different too. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:09:00 -
[52]
Update
I've found evidence of EVE also being affected by this, as I have found cache entries for calls to the same method, with different arguments, that exhibit the same pickling inconsistency:
"(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVX-Factor Ind\nI0\ntp3\n." <- only three PUTs "(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVNTT\np3\nI0\ntp4\n." <- four PUTs
So, it's pretty much impossible to know whether EVE held more than 1 reference to a string when it hashed the cache key, and as such there's no good way to solve this problem.
Recursively collecting all elements of the key into a new list so that their reference count is going to be at least 2 before using GetCacheFileName() does seem to fix it for most of the cache entries, but there's still going to be a handful that fail to produce the correct hash.
Try the following workaround, but note that you should only do this for the CachedMethodCalls folder.
def collect(key): foo = [] _collect((key,), foo.append) return foo
def _collect(items, append): for element in items: if type(element) in (list, tuple): _collect(element, append) append(element)
foo = collect(key) filename = cm.GetCacheFileName(key)
Guess I'll be poking some people in CCP about this.
_
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2010.08.03 18:22:00 -
[53]
Originally by: Entity Guess I'll be poking some people in CCP about this.
Please follow the normal procedure and explain the bug to a bughunter first. --
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:55:00 -
[54]
Originally by: Catari Taga
Originally by: Entity Guess I'll be poking some people in CCP about this.
Please follow the normal procedure and explain the bug to a bughunter first.
_
Got Item? | EVE API? | Cache? |
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2010.08.03 20:22:00 -
[55]
Pam-pam-pam... This could likely be the case for "stuck" corporation assets listings. -- Thanks CCP for cu |
SmallBrain
|
Posted - 2010.08.03 21:58:00 -
[56]
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
A bug still being a bug, the filename of course should exist, but I solved this differently. This snippet might help other people too:
def LoadOurCacheFolder(cache, name): crap = {} if cache.machoVersion > 181 and name.lower() == "bulkdata": name = cache.bulkdatapath else: name = os.path.join(cache.machocachepath, name) for filename in glob.glob(os.path.join(name, "*.cache")): try: what, obj = blue.marshal.Load(open(filename, "rb").read()) except IOError: continue obj['filename'] = filename crap[what] = obj return crap
This assigns the filename to the object too. Then I just do:
cmc = LoadOurCacheFolder(cache, "CachedMethodCalls") for key, obj in cmc.iteritems(): os.unlink(obj['filename'])
Well .. the unlink is done when I know for sure the data is posted, but you get the idea :) So now the globl always returns new files. And a nice side-effect: it keeps the cache-dirs relative empty, and keep it from consuming tons of MiB when you scan the complete market in a few regions ;)
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 22:21:00 -
[57]
yep, that works too.
I've send off some mails to the right people so hopefully the bug is going to be fixed sometime SoonÖ and i'll update Reverence :)
_
Got Item? | EVE API? | Cache? |
Saphira Khan
The Wretched.
|
Posted - 2010.08.11 17:56:00 -
[58]
Thanks for the help all, will try this soon! www.eve-lotteries.com |
Induc
Amarr
|
Posted - 2010.08.17 20:10:00 -
[59]
Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.18 12:32:00 -
[60]
Originally by: Induc Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?
Working on it. It compiles for 2.7 it seems. Will have to check EVE first once it's up, to see if the marshal format and cache key hashing are still the same.
Update SoonÖ _
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.21 13:08:00 -
[61]
Hm well I couldn't find any problems with the 2.7 build.
git updated, I'll put a 2.7 binary distribution SoonÖ _
Got Item? | EVE API? | Cache? |
Lady Rensa
|
Posted - 2010.09.02 23:38:00 -
[62]
Originally by: Entity I'll put a 2.7 binary distribution SoonÖ
This would be great, thank you. |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.09.04 11:43:00 -
[63]
2.6 and 2.7 binaries now available.
Fixed some stuff as well: - Reverence now attempts to import modules from its own package folder first before trying a global import to avoid conflicts with other packages. - Automatic cache path discovery now slightly more intelligent and won't be fooled by an empty cache folder in client installs when there's one in %LOCALAPPDATA%.
_
Got Item? | EVE API? | Cache? |
Xendrais
|
Posted - 2010.10.10 02:20:00 -
[64]
Very nice library
Since the last patch 1.1, I get the following error:
Traceback (most recent call last): File "<stdin>", line 4, in <module> File "C:\Python27\lib\site-packages\reverence\config.py", line 376, in __get__ value = self.method(obj) File "C:\Python27\lib\site-packages\reverence\config.py", line 388, in method return self._loadbulkdata(tableName=(tableName or attrName), storageClass=storageClass, rowClass=rowClass, primaryKey=primaryKey) File "C:\Python27\lib\site-packages\reverence\config.py", line 740, in _loadbulkdata obj = self.cache.LoadObject(fullTableName) File "C:\Python27\lib\site-packages\reverence\cache.py", line 285, in LoadObject raise RuntimeError("cache file not found: %s" % fileName) RuntimeError: cache file not found: ce99.cache
This file "ce99.cache" exists in CachedObjects but not in CachedMethodCalls.
Any idea why and suggestions to correct?
Thanks
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.13 17:18:00 -
[65]
Originally by: Xendrais Very nice library
Since the last patch 1.1, I get the following error:
Traceback (most recent call last): File "<stdin>", line 4, in <module> File "C:\Python27\lib\site-packages\reverence\config.py", line 376, in __get__ value = self.method(obj) File "C:\Python27\lib\site-packages\reverence\config.py", line 388, in method return self._loadbulkdata(tableName=(tableName or attrName), storageClass=storageClass, rowClass=rowClass, primaryKey=primaryKey) File "C:\Python27\lib\site-packages\reverence\config.py", line 740, in _loadbulkdata obj = self.cache.LoadObject(fullTableName) File "C:\Python27\lib\site-packages\reverence\cache.py", line 285, in LoadObject raise RuntimeError("cache file not found: %s" % fileName) RuntimeError: cache file not found: ce99.cache
This file "ce99.cache" exists in CachedObjects but not in CachedMethodCalls.
Any idea why and suggestions to correct?
Thanks
Reverence might be confused about the location of the cache (can happen if you use /LUA:OFF). Also, it seems that one of the patches cleaned out the client's BulkData folder. Which means that if you have not logged in properly at least once, reverence would give that sort of error. So, does it still fail? As my copy seems to work just fine.
_
Got Item? | EVE API? | Cache? |
Xendrais
|
Posted - 2010.10.14 14:28:00 -
[66]
Originally by: Entity
Reverence might be confused about the location of the cache (can happen if you use /LUA:OFF). Also, it seems that one of the patches cleaned out the client's BulkData folder. Which means that if you have not logged in properly at least once, reverence would give that sort of error. So, does it still fail? As my copy seems to work just fine.
Thanks for your time.
I do not use the /LUA:OFF switch.
I checked and found two things:
1) The file ce99.cache is sometimes like 1,059 Kbytes long and sometimes less than 1 Kb long. Reverence work ok with the 1,059 Kb file. I believe it means the file gets overwritten sometimes.
2) Also under MachoNet, there are now two groups of folders: 87.237.38.200\247 and 127.0.0.1\249
The later group is the most current one. Reverence does not find this group but only the first one! Thus giving old information...
Any more idea?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.14 18:17:00 -
[67]
Originally by: Xendrais 2) Also under MachoNet, there are now two groups of folders: 87.237.38.200\247 and 127.0.0.1\249
The later group is the most current one. Reverence does not find this group but only the first one! Thus giving old information...
Any more idea?
127.0.0.1? what? are you running EVE through a local proxy or something? You'll have to use eve=blue.EVE(server="127.0.0.1") to access the data from there then.
_
Got Item? | EVE API? | Cache? |
Xendrais
|
Posted - 2010.10.15 20:08:00 -
[68]
Originally by: Entity
127.0.0.1? what? are you running EVE through a local proxy or something? You'll have to use eve=blue.EVE(server="127.0.0.1") to access the data from there then.
Thanks for the tip.
No proxy in use as far as I know!!!
|
Induc
Amarr
|
Posted - 2010.10.15 22:47:00 -
[69]
Originally by: Xendrais Edited by: Xendrais on 15/10/2010 21:14:58
Originally by: Entity
127.0.0.1? what? are you running EVE through a local proxy or something? You'll have to use eve=blue.EVE(server="127.0.0.1") to access the data from there then.
Thanks for the tip. Did try it but received error:
>>> eve = blue.EVE(server="127.0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: __init__() takes at least 2 arguments (2 given) >>>
And no proxy in use as far as I know!!!
You need to provide the path to your eve installation too. >>> eve = blue.EVE("C:/...", server="127.0.0.1")
|
Xendrais
|
Posted - 2010.10.15 23:54:00 -
[70]
Originally by: Induc
You need to provide the path to your eve installation too. >>> eve = blue.EVE("C:/...", server="127.0.0.1")
Now I get this and the path to EVE is good:
>>> eve = blue.EVE("C:\Program Files\CCP\EVE", server="127.0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python27\lib\site-packages\reverence\blue.py", line 130, in __init__ self.cache = cache.CacheMgr(self.root, self.server, machoVersion, cachepath)
File "C:\Python27\lib\site-packages\reverence\cache.py", line 218, in __init__ raise RuntimeError("Could not determine EVE cache folder location.") RuntimeError: Could not determine EVE cache folder location. >>>
|
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.16 01:40:00 -
[71]
Originally by: Xendrais
Originally by: Induc
You need to provide the path to your eve installation too. >>> eve = blue.EVE("C:/...", server="127.0.0.1")
Now I get this and the path to EVE is good:
>>> eve = blue.EVE("C:\Program Files\CCP\EVE", server="127.0.0.1") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python27\lib\site-packages\reverence\blue.py", line 130, in __init__ self.cache = cache.CacheMgr(self.root, self.server, machoVersion, cachepath)
File "C:\Python27\lib\site-packages\reverence\cache.py", line 218, in __init__ raise RuntimeError("Could not determine EVE cache folder location.") RuntimeError: Could not determine EVE cache folder location. >>>
Hrm, well, I think you're better off trying to figure out why you are getting a 127.0.0.1 folder in your cache instead of the tranquility IP. That's the real problem here. If you know how to reproduce this situation, let me know. I'm not getting a 127.0.0.1, so I don't know what is going on, and Reverence can't magically figure out where your files are in non-standard situations like yours.
The only thing I could think of as a workaround is providing cachepath=r"C:\path\to\eve's\cache\" to blue.EVE() as well. If you're not using LUA and are on XP, it would be r"C:\Documents and Settings\USERNAME\Local Settings\Application Data\CCP\EVE\mangled_copy_of_install_path\cache". Forgot where it is on Win7 but i'm sure you can find it :P
(On a side note, it's a good idea not to use backslashes bare in python strings. Escape them, use the r string prefix, or use forward slashes to avoid problems)
_
Got Item? | EVE API? | Cache? |
D Spectre
Amok. Goonswarm Federation
|
Posted - 2010.10.23 06:29:00 -
[72]
I'll start out by saying i know nothing about python but i am really wanting to read the market cache and the market graph cache. That being said I've spent at least 8 hours over the last 2 days trying to get something to work. the libevecache dumper works fine for market orders but i would like to be able to parse out the market graph. i've been struggling through this code for hours upon hours trying to get it to work. That being said its probably because i'm an idiot.... I have done a ton of different things trying to make this work but the most recent pretty much sums it up... I take the code located here: http://github.com/ntt/reverence/blob/master/examples/datadump.py change cachedObjects2 = c.LoadCacheFolder("CachedObjects") to cachedObjects2 = c.LoadCacheFolder("CachedMethodCalls")
whenever i try to run it i get the following: AttributeError: 'dict' object has no attribute 'objectID'
(overall i have no clue how to access the attributes of the object imported for methodcalls and make any meaning of them).
And... sorry for being a newb.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.24 11:28:00 -
[73]
Originally by: D Spectre blah
Well yeah modifying the example to use CMC isn't going to work because CachedMethodCalls are different objects.
try something like this:
from reverence import blue
eve = blue.EVE("C:/path/to/eve") cm = eve.getcachemgr() crap = cm.LoadCacheFolder("CachedMethodCalls")
for key, obj in crap.iteritems(): if "GetOrders" in key: orders = obj['lret'] for order in orders[0]: # 0 for sell, 1 for buy orders print order.orderID, order.price, int(order.volRemaining)
_
Got Item? | EVE API? | Cache? |
Tanmoe
|
Posted - 2010.10.27 12:23:00 -
[74]
I'm unable to find the documentation for the fields in the orders records. Most are obvious but i'm missing a few things.
for example : jumps don't appear to be right for buy orders. What is the issued value?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.10.27 14:43:00 -
[75]
Originally by: Tanmoe I'm unable to find the documentation for the fields in the orders records. Most are obvious but i'm missing a few things.
for example : jumps don't appear to be right for buy orders. What is the issued value?
I believe it is the server's idea of distance. The client calculates jump distances with your current autopilot settings (which are entirely client-side) before displaying them in the market view anyway.
You should probably ignore that field and calculate it yourself if you require that info.
_
Got Item? | EVE API? | Cache? |
Xendrais
|
Posted - 2010.10.29 14:27:00 -
[76]
Originally by: Entity
Hrm, well, I think you're better off trying to figure out why you are getting a 127.0.0.1 folder in your cache instead of the tranquility IP. That's the real problem here.
If you know how to reproduce this situation, let me know. I'm not getting a 127.0.0.1, so I don't know what is going on, and Reverence can't magically figure out where your files are in non-standard situations like yours. ...
Well, just to let you know, EVE decided (sometime this week prior to patch 1.1.6 and on its own) to revert to using the ...\cache\MachoNet\87.237.38.200\249 folder again. Don't know what did it!!! and I cannot reproduce it.
|
T'Senthor
|
Posted - 2010.11.05 10:27:00 -
[77]
Is there anything equivalent to this for .NET?
I considered trying to run it using IronPython, but then I noticed that half of it is implemented in c, so that's not going to fly.
|
Sigmaz
United Systems Navy Wildly Inappropriate.
|
Posted - 2010.11.15 17:11:00 -
[78]
Originally by: T'Senthor Is there anything equivalent to this for .NET?
I considered trying to run it using IronPython, but then I noticed that half of it is implemented in c, so that's not going to fly.
A pure python version would be very much appreciated for this reason.
|
Cecil Eckard
|
Posted - 2010.12.09 18:09:00 -
[79]
I'm porting this to Java at the moment but I'm having trouble with TYPE_STREAM 0x2b.
The code seems to indicate that the next byte is the length of the stream - unless it's 0xff, in which case the next four bytes form an integer with the real stream length.
The trouble is I'm getting 0x2b 0xff 0x1e 0x04 0x00 0x00 0x7e. The last byte makes me think I'm still in the right place, but the previous four bytes are giving me ridiculously big ints.
Am I interpreting the byte sequence incorrectly, or is there another way to determine the end of the substream?
|
Jay Wareth
|
Posted - 2010.12.11 08:24:00 -
[80]
Edited by: Jay Wareth on 11/12/2010 08:30:13 With the pending demise of eve-metrics I set out to find a new data source for my spreadsheets, took me about 2 hours to write a little script to read my own cache for all the price data I need.
Thanks for this awesome tool.
Edit: Apparently I am not allowed to (sc)**** the cache
|
|
Enilonee
|
Posted - 2010.12.13 14:18:00 -
[81]
Originally by: Cecil Eckard I'm porting this to Java at the moment
I'd love to see this done! Keep us posted!
|
9yards
Caldari Eve Engineering Finance Eve Engineering
|
Posted - 2010.12.13 16:18:00 -
[82]
Has anyone had any luck getting Reverence to compile to a single exe? I'm creating a little analysis tool that does some math on the price history data, but I'd like to make it so that an install of Python isn't necessary to use it. A .NET version would of course be perfect. I'd be willing to work on something like that but I'm struggling with how to get started reading the data. I'm not python fluent at all so understanding that code has got me spinning my wheels alot.
Eve Engineering Lottery Eve Engineering Finance
CEO, Founder |
Muscaat
ASCENTIS CORPORATION Free Worlds Alliance
|
Posted - 2010.12.13 18:15:00 -
[83]
Originally by: Cecil Eckard I'm porting this to Java at the moment...
Cecil - feel free to plagiarise any of my work on a Java cache reader - source at this link ---------- EVE Markets - global price trends at your fingertips |
Mella Elcus
|
Posted - 2010.12.14 14:17:00 -
[84]
Edited by: Mella Elcus on 14/12/2010 14:17:33
Originally by: 9yards Has anyone had any luck getting Reverence to compile to a single exe? I'm creating a little analysis tool that does some math on the price history data, but I'd like to make it so that an install of Python isn't necessary to use it. A .NET version would of course be perfect. I'd be willing to work on something like that but I'm struggling with how to get started reading the data. I'm not python fluent at all so understanding that code has got me spinning my wheels alot.
You could use py2exe to generate a single .exe of a python script that prints various cache data using Reverence, and then parse it in .NET. It would still be Python though, just wrapped into a single (quite large) .exe.
|
9yards
Caldari Eve Engineering Finance Eve Engineering
|
Posted - 2010.12.14 19:23:00 -
[85]
I did try py2exe before i posted and never could get a successful compile.
Eve Engineering Lottery Eve Engineering Finance
CEO, Founder |
Cecil Eckard
|
Posted - 2010.12.16 09:24:00 -
[86]
Originally by: Muscaat
Originally by: Cecil Eckard I'm porting this to Java at the moment...
Cecil - feel free to plagiarise any of my work on a Java cache reader - source at this link
Typical, someone's already beat me to it.
|
Papa Yoru
Gallente Electric Monk inc United Rock Enterprises Ltd
|
Posted - 2010.12.16 22:58:00 -
[87]
Originally by: 9yards I did try py2exe before i posted and never could get a successful compile.
The archive library package created when building with py2exe doesn't grab all the reverence stuff, you can manually add the missing stuff though. +++ Reality Error 404 - Reboot Cosmos +++ |
Gemini Rock
|
Posted - 2010.12.17 04:42:00 -
[88]
I mean this in the nicest possible way, but does anyone speak english around here?
So, I've downloaded and installed Python2.7 .... woo.
I've downloaded and run the installer for Reverence .... go me!
What happens next? Have I missed a really simple instruction somewhere. Seriously, I could be being really thick here but how do I use this?
|
Two step
Amarr Aperture Harmonics K162
|
Posted - 2010.12.17 17:24:00 -
[89]
Originally by: Gemini Rock I mean this in the nicest possible way, but does anyone speak english around here?
So, I've downloaded and installed Python2.7 .... woo.
I've downloaded and run the installer for Reverence .... go me!
What happens next? Have I missed a really simple instruction somewhere. Seriously, I could be being really thick here but how do I use this?
Um, what were you expecting to have happen? This is an advanced tool that can be used by 3rd party developers to extract information from the cached eve client data. There are a couple of example scripts in the reverence install.
|
Jay Wareth
|
Posted - 2010.12.18 04:27:00 -
[90]
Originally by: Gemini Rock
What happens next? Have I missed a really simple instruction somewhere. Seriously, I could be being really thick here but how do I use this?
As Two Step said, this isn't an end user tool. It is a tool designed to help developers access EVE's internal data. If you don't know what to do from here you don't have the skills for this tool to be useful to you.
|
|
Dragonaire
Caldari Corax. SOUL CARTEL
|
Posted - 2010.12.18 04:53:00 -
[91]
I might also suggest if you know more about MySQL databases than programming trying Yapeal which does the whole API raw XML to some simple DB tables stuff for you. You might find it a little esaier to understand in that case but it's still not an end user application just a different approach to getting the data into what most people consider an easier form to work with. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2010.12.18 05:37:00 -
[92]
Originally by: Dragonaire I might also suggest if you know more about MySQL databases than programming trying Yapeal which does the whole API raw XML to some simple DB tables stuff for you. You might find it a little esaier to understand in that case but it's still not an end user application just a different approach to getting the data into what most people consider an easier form to work with.
Reverence is unrelated to the EVE API. --
|
Dragonaire
Caldari Corax. SOUL CARTEL
|
Posted - 2010.12.18 15:24:00 -
[93]
Sorry I forgot it's for the cache stuff Not for the first time I've done that just the first time I did not catch myself before posting -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Manathera
|
Posted - 2010.12.18 21:12:00 -
[94]
Edited by: Manathera on 18/12/2010 21:13:13 I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable. MarketReader Please let me know of any errors as this is the first python code I've ever written.
|
Theodor Bliesheimer
Integrated Astromechanics Corp. Zeta Conglomerate
|
Posted - 2010.12.19 12:04:00 -
[95]
Originally by: Jay Wareth
Originally by: Gemini Rock
What happens next? Have I missed a really simple instruction somewhere. Seriously, I could be being really thick here but how do I use this?
As Two Step said, this isn't an end user tool. It is a tool designed to help developers access EVE's internal data. If you don't know what to do from here you don't have the skills for this tool to be useful to you.
Is there anyone planning to release a Reverence-based end-user compatible tool usable to extract market history data, or has such a tool already been released (with the exception of EveMetrics)?
|
Ix Forres
Caldari Righteous Chaps
|
Posted - 2010.12.20 01:07:00 -
[96]
Originally by: Theodor Bliesheimer
Originally by: Jay Wareth
Originally by: Gemini Rock
What happens next? Have I missed a really simple instruction somewhere. Seriously, I could be being really thick here but how do I use this?
As Two Step said, this isn't an end user tool. It is a tool designed to help developers access EVE's internal data. If you don't know what to do from here you don't have the skills for this tool to be useful to you.
Is there anyone planning to release a Reverence-based end-user compatible tool usable to extract market history data, or has such a tool already been released (with the exception of EveMetrics)?
https://github.com/JamesHarrison/evemetrics_uploader is open source, Reverence-based, and free for people to fork into their own projects or steal from as much as they want to. Go wild. -- Ix Forres - 3rd Party Application Developer - EVE Metrics - accVIEW
|
Blazing Fire
Interstellar Operations Incorporated
|
Posted - 2010.12.30 22:32:00 -
[97]
Hi,
From the readme it is not clear to me if this tool read only the static data or it can be used to read the market orders and market history. Can someone confirm.
I am a newbie .Net codder and I want to read market data in my app. and I have no idea if this Pythin library can be used on .Net app and how.
Is there a place where I can read about the actual structure of the EVE .cache files? Again I am interested in the market data.
TIA, Blaze
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.19 18:53:00 -
[98]
Originally by: Blazing Fire From the readme it is not clear to me if this tool read only the static data or it can be used to read the market orders and market history. Can someone confirm.
I am a newbie .Net codder and I want to read market data in my app. and I have no idea if this Pythin library can be used on .Net app and how.
Is there a place where I can read about the actual structure of the EVE .cache files? Again I am interested in the market data.
It can read anything in the cache. which includes market orders/history as that stuff appears in the cache when you look it up ingame.
Reverence is not compatible with IronPython afaik (so much for .NET)
No documentation for specific cache objects exists.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.19 18:58:00 -
[99]
Notice to Reverence users:
It seems that with this patch, one or more of the static data tables died.
config.BulkData.ramtypematerials is one of the casualties, as the bill of materials seems to be created serverside by request now, judging by the new GetMaterialsForTypeWithActivity method I see in my cache.
This means cfg.prime() will fail (funnily enough on the very last table it loads), but apps using transparent loading that do not use the ramtypematerials table will continue to function as normal.
I'll update the library to not load ramtypematerials in cfg.prime() soonÖ
_
Got Item? | EVE API? | Cache? |
Takeshi Ryuu
Infernal laboratory Blind Octopus
|
Posted - 2011.01.20 15:05:00 -
[100]
What am i doing wrong?
Quote: >>> cmc = cachemgr.LoadCacheFolder("CachedMethodCalls") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "D:\tools\python-2.7.1\lib\site-packages\reverence\cache.py", line 231, in LoadCacheFolder what, obj = blue.marshal.Load(open(filename,"rb").read()) AttributeError: 'module' object has no attribute 'CFilterRowset'
cachemgr is initialized the same way as in examples posted on previous pages.
|
|
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2011.01.20 15:17:00 -
[101]
Originally by: Entity Notice to Reverence users:
It seems that with this patch, one or more of the static data tables died.
config.BulkData.ramtypematerials is one of the casualties,
Hmm, I haven't looked into new cache structure at all yet but my cache to datadump converter still pulls that table just fine using (an old version of) your library. --
|
Two step
Aperture Harmonics K162
|
Posted - 2011.01.20 16:57:00 -
[102]
Originally by: Entity Notice to Reverence users:
It seems that with this patch, one or more of the static data tables died.
config.BulkData.ramtypematerials is one of the casualties, as the bill of materials seems to be created serverside by request now, judging by the new GetMaterialsForTypeWithActivity method I see in my cache.
This means cfg.prime() will fail (funnily enough on the very last table it loads), but apps using transparent loading that do not use the ramtypematerials table will continue to function as normal.
I'll update the library to not load ramtypematerials in cfg.prime() soonÖ
I'm pretty sure it just got renamed to invtypematerials or something similar. I was looking at diffs between tq (pre-patch) and sisi with the final patch and that is what I saw.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.20 18:35:00 -
[103]
Originally by: Two step I'm pretty sure it just got renamed to invtypematerials or something similar. I was looking at diffs between tq (pre-patch) and sisi with the final patch and that is what I saw.
Ah, I must've read invtypereactions, I'm blind. Thanks :P A table name change still requires a reverence update though :-/
_
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2011.01.20 21:33:00 -
[104]
Originally by: Two step I'm pretty sure it just got renamed to invtypematerials or something similar. I was looking at diffs between tq (pre-patch) and sisi with the final patch and that is what I saw.
Oh yes that's it. My code always renamed it anyway so I didn't notice. :) --
|
6ie
|
Posted - 2011.01.20 23:07:00 -
[105]
Originally by: Takeshi Ryuu What am i doing wrong?
Quote: >>> cmc = cachemgr.LoadCacheFolder("CachedMethodCalls") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "D:\tools\python-2.7.1\lib\site-packages\reverence\cache.py", line 231, in LoadCacheFolder what, obj = blue.marshal.Load(open(filename,"rb").read()) AttributeError: 'module' object has no attribute 'CFilterRowset'
cachemgr is initialized the same way as in examples posted on previous pages.
hmmm i am getting this error too now. It worked prior to the Tuesdays patch. I have multiple sub-dirs under \MachoNet\87.237.38.200\ No idea if that is relevant.
|
Covert Kitty
Amarr ISK Solutions SRS.
|
Posted - 2011.01.30 07:27:00 -
[106]
I get that error as well, any idea whats causing it?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.01.30 07:56:00 -
[107]
CCP's moving the handling of certain types of data structures from Python into C++ for performance. So, CFilterRowset is the C++ implementation of the python FilterRowset, and I have yet to write a class for it. I haven't looked at the structure yet, will do SoonÖ
_
Got Item? | EVE API? | Cache? |
Da Trader
|
Posted - 2011.01.30 15:44:00 -
[108]
Sorry to revive topic with stupid Q, but does it still work after new patch?
|
Covert Kitty
Amarr ISK Solutions SRS.
|
Posted - 2011.01.31 11:13:00 -
[109]
parts of it do, the market cache doesnt seem to however due to the error above
|
6ie
|
Posted - 2011.01.31 21:06:00 -
[110]
Let us all send Entity 100mil to complete that code. I will be 2nd, go.
|
|
Theodor Bliesheimer
Integrated Astromechanics Corp. Zeta Conglomerate
|
Posted - 2011.02.01 05:32:00 -
[111]
One of my corp members fixed his Reverence errors after the latest patch by deleting his cache folder (C:\Users\XXX\AppData\Local\CCP\EVE\c_eve-online_tranquility\cache\). Not sure if that will work for everyone but it's at least worth a try.
|
Verite Rendition
Caldari F.R.E.E. Explorer The Initiative.
|
Posted - 2011.02.02 02:16:00 -
[112]
Now that I have some free time, I have a chance to look in to this.
Can anyone tell me if Reverence can extract mapsoloarsystems and other such data from EVE? I'm looking at what it would take to do a periodic version of the Influence map for Serenity (EVE-China); since they don't have an API I'd need to extract sovereignty information, stations, and an alliance list from the client. Is this possible with Reverence? ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2011.02.02 03:10:00 -
[113]
Originally by: Verite Rendition Now that I have some free time, I have a chance to look in to this.
Can anyone tell me if Reverence can extract mapsoloarsystems and other such data from EVE? I'm looking at what it would take to do a periodic version of the Influence map for Serenity (EVE-China); since they don't have an API I'd need to extract sovereignty information, stations, and an alliance list from the client. Is this possible with Reverence?
You can pull everything that you can see in the client, but to get the data you will need to request it via the client first, after that the information will be cached and can be accessed via Reverence.
This will be easy for some data (Sov data, development indices, list of alliances themselves, but for things like alliance members you'll need to show info on each of them (the ingame browser and some javascript can help here). I'm not sure whether you can get a conquerable stations list in game so that data might not be available (short of doing a show info on every conquerable solar system and then on the listed stations). --
|
Verite Rendition
Caldari F.R.E.E. Explorer The Initiative.
|
Posted - 2011.02.02 03:21:00 -
[114]
Edited by: Verite Rendition on 02/02/2011 03:24:10
Originally by: Catari Taga
Originally by: Verite Rendition Now that I have some free time, I have a chance to look in to this.
Can anyone tell me if Reverence can extract mapsoloarsystems and other such data from EVE? I'm looking at what it would take to do a periodic version of the Influence map for Serenity (EVE-China); since they don't have an API I'd need to extract sovereignty information, stations, and an alliance list from the client. Is this possible with Reverence?
You can pull everything that you can see in the client, but to get the data you will need to request it via the client first, after that the information will be cached and can be accessed via Reverence.
This will be easy for some data (Sov data, development indices, list of alliances themselves, but for things like alliance members you'll need to show info on each of them (the ingame browser and some javascript can help here). I'm not sure whether you can get a conquerable stations list in game so that data might not be available (short of doing a show info on every conquerable solar system and then on the listed stations).
Thanks for the quick reply Catari.
Just digging through the code and using Entity's datadump example, it looks like I can get most of the information I need. Once the client has pulled up the in-game sov map and in-game station map, the cached data becomes available as MethodCall.server.stationSvc.GetAllianceSystems and MethodCall.server.map.GetStationCount respectively. Both of which are a bit weird, but it's enough data to work with.
The remaining issue seems to be that I can't find a list of the alliances (even after having the client open up the alliance ranking list). I don't need to know the individual corps for now, but I can't even get a complete list of alliances and their names. Looking through the cache resources on EVE-Dev, I'm not even sure there's a proper list in the cache, is there?
And while I'm on a note about weird things, looking at MethodCall.server.devIndexManager.GetAllDevelopmentIndices, it's interesting to note that it uses the raw point value instead of the calculated level. Has anyone ever figured out how many points correlate to which level? ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2011.02.02 03:40:00 -
[115]
See MethodCall.server.allianceRegistry.GetRankedAlliances for the alliance list.
As to point to level conversion on system upgrades, have never looked into that and can't help there, but you'll figure it out quickly just by comparing how it's displayed in game. --
|
Verite Rendition
Caldari F.R.E.E. Explorer The Initiative.
|
Posted - 2011.02.02 05:25:00 -
[116]
Originally by: Catari Taga See MethodCall.server.allianceRegistry.GetRankedAlliances for the alliance list.
What's the trick to getting that to come out of the cache? Even after the client has accessed it, the datadump script isn't handing it over. ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2011.02.02 12:11:00 -
[117]
Originally by: Verite Rendition
Originally by: Catari Taga See MethodCall.server.allianceRegistry.GetRankedAlliances for the alliance list.
What's the trick to getting that to come out of the cache? Even after the client has accessed it, the datadump script isn't handing it over.
I had to look at Entity's example script for that (the things I do ). If you comment out lines 94 and 96 of that and it will dump it as MethodCall.server.allianceRegistry.GetRankedAlliances.0.sql. --
|
Axemaster
|
Posted - 2011.02.02 23:53:00 -
[118]
Hi, I was wondering when this might be working for mac OS-X 10.6? I tried to get the basic cache reading to work but had absolutely no luck, and the python libraries weren't installing correctly.
I have been searching fruitlessly for quite some time... there doesn't seem to be any useful info anywhere regarding how I could decrypt this stuff. Very frustrating.
As far as I can tell, the caches are encrypted using a python function called "marshal", but I have no understanding of it. It seems to be a black box, with no published documentation that I can find.
I was hoping that someone might be able to post the decrypting code in a sort of "general format" (the full code, with no "modules" or language-specific functions) so I could translate it easily into another programming language... I don't need any fancy stuff, just the basic code that converts the cache files into text.
|
Kitty Kupcake
|
Posted - 2011.02.03 02:54:00 -
[119]
Originally by: Theodor Bliesheimer One of my corp members fixed his Reverence errors after the latest patch by deleting his cache folder (C:\Users\XXX\AppData\Local\CCP\EVE\c_eve-online_tranquility\cache\). Not sure if that will work for everyone but it's at least worth a try.
To be more specific, you need to remove the folders under the directory cache\MachoNet\###\
Where ### is some folder with a number as a name. Example 12.345.67.890
And of course before you do that, please make a backup.
---
So to reiterate, I was having issues with the following line of code:
cmc = cachemgr.LoadCacheFolder("CachedMethodCalls")
To fix it, I removed the folders under the directory:
C:\Users\XXX\AppData\Local\CCP\EVE\c_eve-online_tranquility\cache\MachoNet\###\
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.03 05:49:00 -
[120]
If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache. _
Got Item? | EVE API? | Cache? |
|
Verite Rendition
Caldari F.R.E.E. Explorer The Initiative.
|
Posted - 2011.02.03 14:52:00 -
[121]
Originally by: Catari Taga
Originally by: Verite Rendition
Originally by: Catari Taga See MethodCall.server.allianceRegistry.GetRankedAlliances for the alliance list.
What's the trick to getting that to come out of the cache? Even after the client has accessed it, the datadump script isn't handing it over.
I had to look at Entity's example script for that (the things I do ). If you comment out lines 94 and 96 of that and it will dump it as MethodCall.server.allianceRegistry.GetRankedAlliances.0.sql.
That did it. Thanks, you're my hero! ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Schry Evolon
Sebiestor Tribe
|
Posted - 2011.02.04 00:54:00 -
[122]
Originally by: Entity If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.
Assuming you are "ntt" of the GitHub, I have sent it to your email addy.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.04 07:43:00 -
[123]
Originally by: Schry Evolon
Originally by: Entity If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.
Assuming you are "ntt" of the GitHub, I have sent it to your email addy.
Hmm that one works here. I guess it's one of the smaller changes I haven't put into a release yet. _
Got Item? | EVE API? | Cache? |
Schry Evolon
Sebiestor Tribe
|
Posted - 2011.02.04 21:07:00 -
[124]
Originally by: Entity
Originally by: Schry Evolon
Originally by: Entity If anyone wants to give me one of the files that fails to decode on the CFilterRowset... I don't seem to have any of them in my cache.
Assuming you are "ntt" of the GitHub, I have sent it to your email addy.
Hmm that one works here. I guess it's one of the smaller changes I haven't put into a release yet.
Emailed you again.
If I can help, I would like to assist and test the unpublished updates to see if the resolution is within.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.02.07 09:19:00 -
[125]
I've committed a potential fix for the CFilterRowset problem. Update and build from source if you want to test it.
Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.
Guess you'll find out soon enough ;) _
Got Item? | EVE API? | Cache? |
Schry Evolon
Sebiestor Tribe
|
Posted - 2011.02.08 00:11:00 -
[126]
Originally by: Entity I've committed a potential fix for the CFilterRowset problem. Update and build from source if you want to test it.
Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.
Guess you'll find out soon enough ;)
Everything works in the new build, so far.
|
Verite Rendition
Caldari F.R.E.E. Explorer The Initiative.
|
Posted - 2011.02.08 04:59:00 -
[127]
If it's of any use to you Entity, Reverence works fine on Serenity (China) once you patch the cache directory locations.
Default client location: C:\Program Files\Optic\EVE
Default cache directory location: ...\CCP\EVE\c_program_files_optic_eve_tel.eve.gtgame.com.cn
MachoNet version is 219. They're still on Apocrypha (6.14.179118). ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Nikolai Kondratiev
Sphere Design Inc.
|
Posted - 2011.02.13 10:30:00 -
[128]
Originally by: Entity I've committed a potential fix for the CFilterRowset problem. Update and build from source if you want to test it.
Note that I don't actually have any such files that contain a populated table, so I don't know if it will work on those.
Guess you'll find out soon enough ;)
I tried that new build (pretty sure I compiled new build correctly, python n00b here) with some cache file that would cause the CFilterRowset error. But instead of returning an error, python (windows version 2.7) just crashes when loading CachedMethodCalls on a test script.
Any idea on what could cause the crash ? _ Ore Table | PI Profits |
Papa Yoru
Gallente Electric Monk inc United Rock Enterprises Ltd
|
Posted - 2011.02.13 16:22:00 -
[129]
Not sure if this is helpful, but EVE-Centrals uploader also fails now with the CFilterRowset issue. It is definitely something new since Incursions patchs. The only workaround i can find is to clear my cache each time I encounter this. +++ Reality Error 404 - Reboot Cosmos +++ |
Thart
U.K.R.A.I.N.E SOLAR FLEET
|
Posted - 2011.03.10 10:18:00 -
[130]
Users of my program reports problem with cache reader based on Reverence.
Quote: [error] Updating market orders (cache)... [error] Error calling cache import tool. Code: 255
Quote: Traceback (most recent call last): File "evemarketordersfromcache.py", line 49, in <module> AttributeError: 'module' object has no attribute 'StackSize'
I got this problem too. Here is a two files from eve online cache whih cause a problem - link ----------------------------------------- EVE Mentat - true trade tool. |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.10 12:00:00 -
[131]
Originally by: Thart Edited by: Thart on 10/03/2011 10:37:25 Users of my program reports problem with cache reader based on Reverence.
Quote: [error] Updating market orders (cache)... [error] Error calling cache import tool. Code: 255
Quote: Traceback (most recent call last): File "evemarketordersfromcache.py", line 49, in <module> AttributeError: 'module' object has no attribute 'StackSize'
The changes to the inventory system in latest patch caused this. looking into it. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.10 20:57:00 -
[132]
I've pushed a fix for the inventory system changes. Update/rebuild your copy of Reverence from source to try it out. (No new installer of yet).
Thanks to Thart for providing test files :)
For the technical: The fix was not trivial; CCP gave the DBRow subsystem a new feature that basically implements Python properties.
_
Got Item? | EVE API? | Cache? |
Papa Yoru
Gallente The Graduates Morsus Mihi
|
Posted - 2011.03.17 15:59:00 -
[133]
Thanks for the update :)
+++ Reality Error 404 - Reboot Cosmos +++ |
Theodor Bliesheimer
Integrated Astromechanics Corp. Zeta Conglomerate
|
Posted - 2011.03.19 00:31:00 -
[134]
Any estimate when an updated installer will be released?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.19 13:02:00 -
[135]
Originally by: Theodor Bliesheimer Any estimate when an updated installer will be released?
Reason I'm not doing this as frequently as I should (read: every build) is because it's a pain to manually upload files to github. I'll upload new installers when I've finished figuring out their upload mechanics so I can script it :P
_
Got Item? | EVE API? | Cache? |
Zeta Zhul
Caldari Preemptive Paranoia
|
Posted - 2011.03.21 04:09:00 -
[136]
A couple things.
1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.
Is there actually a reason why it deletes every single file regardless of the extension or type?
Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.
2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.
sigh.
3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.03.21 11:47:00 -
[137]
Originally by: Zeta Zhul A couple things.
1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.
Is there actually a reason why it deletes every single file regardless of the extension or type?
Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.
2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.
sigh.
3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.
Wrong thread? _
Got Item? | EVE API? | Cache? |
Zeta Zhul
Caldari Preemptive Paranoia
|
Posted - 2011.03.21 15:29:00 -
[138]
Originally by: Entity
Originally by: Zeta Zhul A couple things.
1. The sample market history script on pastie.com does a walk through all directories, subdirectories, etc and deletes every single file.
Is there actually a reason why it deletes every single file regardless of the extension or type?
Because it seems to me that being that the output is in fact a CSV file that perhaps not deleting every last thing in the directory tree might be a worthwhile consideration.
2. If you are planning on using that sample script be advised that if you output the CSV file to your "C:/" root directory that you will in fact be committing computer suicide.
sigh.
3. No I didn't output to the root directory. But I did output to a directory with some subdirectories that had stuff in it that I would rather have kept. Oh well. Sometimes necessity leads to refactoring crappy code.
Wrong thread?
No actually it's the right thread.
It's this script. I'm sure you're aware of it and probably familiar with it. I just wanted to warn people that the script, as is, will conduct a complete directory walk and delete every single file regardless of type. So if you're going to run this script you must ensure that the output directory doesn't have anything important *and* neither do any subdirectories.
Offending code block:
Quote:
print "Deleting old records... \n" for root, dirs, files in os.walk(OUTPATH, topdown=False): for name in files: os.remove(os.path.join(root, name))
I will fully admit that I should have review the whole thing, line by line, before running it. But as the script merely outputs a CSV file it simply did not occur to me that "Deleting old records..." would actually involve deleting everything including old records and pretty much anything else lying around.
|
Alaya Carrier
|
Posted - 2011.04.18 10:00:00 -
[139]
Edited by: Alaya Carrier on 18/04/2011 10:00:23 Is this project dead like libevecache seems to be?
It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.18 13:44:00 -
[140]
Originally by: Alaya Carrier Edited by: Alaya Carrier on 18/04/2011 10:00:23 Is this project dead like libevecache seems to be?
It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source.
It's not dead. Most of my projects simply require very little to no maintenance, even across patches. _
Got Item? | EVE API? | Cache? |
|
Alaya Carrier
|
Posted - 2011.04.18 19:52:00 -
[141]
Edited by: Alaya Carrier on 18/04/2011 19:53:27
Originally by: Entity
Originally by: Alaya Carrier Edited by: Alaya Carrier on 18/04/2011 10:00:23 Is this project dead like libevecache seems to be?
It's easy for me to build the Python part but it's very messy for me to get VC in the computer where I have EvE and Python installed so I can't really build off the git source.
It's not dead. Most of my projects simply require very little to no maintenance, even across patches.
Well, I made a script that worked for a long time with the previous version. Then a recent EvE patch broke it and it'd show errors with syntax like:
for blah in NewPriceHistory: for history_item in blah['lret']: print format_history_row(history_item)
where it complained about blah['lret'] expecting an integer instead of a string on such associative array subscript.
I uninstalled the library. I downloaded the git current release, rebuilt, installed with no errors and...
... now I still get the same associative array error.
What gives?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.19 21:00:00 -
[142]
Edited by: Entity on 19/04/2011 21:00:45
Originally by: Alaya Carrier
Traceback (most recent call last): File "C:\Python26\EvEDump.py", line 54, in <module> for history_item in blah['lret']: TypeError: string indices must be integers, not str
I uninstalled the library. I downloaded the git current release, rebuilt, installed with no errors and...
... now I still get the same associative array error.
What gives?
I'd wager a guess that the structure of that object changed when they patched. I do not control how CCP represents data in cache. This is not a bug.
_
Got Item? | EVE API? | Cache? |
Portaprince
Babylon Creations
|
Posted - 2011.04.22 17:58:00 -
[143]
Awesome work!!! I now have a .NET version of your cache parser (dumper.exe) working. I will release some code soon. It's really just a wrapper for dumper.exe that allows it to be used in .NET. Named "DumpWrapper" of course! Simply point it at a file (.cache files are best) and it hands you back a datatable, xml file, or csv string of the content. I can only compile it for Windows, but someone else could probably make a version for mac and linux...should they desire to stoop to the level of .NET Of course you know this will open the door to more apps!! I've currently only implemented the marketdata portion, but don't see any reason the rest of dumper functionality couldn't be implemented. Funny thing is I tried 4 different ways to get that data: 1) Read the memory from the OS and decode...(yeah, like i have the time to play low level memory games in .NET) 2) Reverse engineer the cache file / Python code... (again, not so easy from .NET) 3) Intercept the OS window handle and try and pull the control caption (not possible, because it doesn't use OS controls) 4) 15 lines of code to create a process and execute dumper on it while capturing the stdout!!!
|
Sverre Haakonson
Gallente
|
Posted - 2011.04.24 20:57:00 -
[144]
Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.04.25 10:35:00 -
[145]
Edited by: Entity on 25/04/2011 10:35:35
Originally by: Sverre Haakonson Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.
Originally by: Entity
DOWNLOAD The project is available on github here: http://github.com/ntt/reverence
There's a windows installer and source distribution in the downloads section.
_
Got Item? | EVE API? | Cache? |
Sverre Haakonson
Gallente
|
Posted - 2011.04.25 22:49:00 -
[146]
Originally by: Entity Edited by: Entity on 25/04/2011 10:35:35
Originally by: Sverre Haakonson Has someone a binary build for windows? Because of python 2.7 was build with vc2008 and I can only access vc2010 it will not work for me.
Originally by: Entity
DOWNLOAD The project is available on github here: http://github.com/ntt/reverence
There's a windows installer and source distribution in the downloads section.
Yes you are right. Shame on me.
|
Moto Akimoto
|
Posted - 2011.05.03 09:52:00 -
[147]
Just FYI, I finally have Reverence running on my Mac OSx. Yay! I'm not a programmer so it took me a while to figure out.
Thanks Entity, this is good stuff! __________________________________ Hello, I sell Rookie ships in bulk. The minimum order value is a gazillion isk. Inquire within. |
Zeta Zhul
Caldari Preemptive Paranoia
|
Posted - 2011.05.07 20:33:00 -
[148]
Originally by: Moto Akimoto Just FYI, I finally have Reverence running on my Mac OSx. Yay! I'm not a programmer so it took me a while to figure out.
Thanks Entity, this is good stuff!
Hey that's great!
BTW I have an updated python script that includes the typeid in the market history extract. What I do is run through a whole list of market items, then run the extract script and then import it all into a MySQL database. It is a lot simpler to have the typeid on every row rather than having to parse out the name of the file.
Let me know if you want it.
|
Mike'P
|
Posted - 2011.05.16 17:48:00 -
[149]
Hi guys,
(although this will probably only make sense to Entity!)
I've been playing around with this on linux, and noticed that I would sometimes get a segmentation violation running LoadCacheFolder("CachedMethodCalls") from python.
I narrowed it down to one cache file (d03c.cache - it has the string 'charGetNewTransactions' in the header) and playing around with src/blue/marshal.c, I notice that it is something to do with the mapping of shared objects, and in particular, the shared_map used in the STORE/RESERVE_SLOT/UPDATE_SLOT macros in marshal.c.
The 'shared_map' table is an array of int32_t values, and is basically the last thing in the cache file. In marshal.c, the table is initialised by pointing to the last x * 4 bytes in the cache file, where x is the number of shared objects in the file (read from the file itself).
With a hex dumper, I was able to see that the tricky cache file didn't appear to have enough bytes at the end to make up a whole number of int32_t types - eg, where it should have had 8 bytes to give 2 x 4 byte integers, it only had six. Hence, share_map was being parked 2 bytes too early in the file, and was picking up an 0x6969 value (in my case) as the LSB, which made the value of shared_map[0] a large number, and hence made
shared_obj[shared_map[0]]
cause a seg fault.
Anyone else seen this ?
I've got a dirty little hack to work around what I think is going on, but as I don't understand the ins and outs of the data structures, I'm hesitant to push it out there as a solution.
Hopefully that makes sense to someone....
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 02:09:00 -
[150]
Originally by: Mike'P Hi guys,
(although this will probably only make sense to Entity!)
Interesting. Can you put this file somewhere I can download it? _
Got Item? | EVE API? | Cache? |
|
Risingson
Mezzanine Inc
|
Posted - 2011.05.17 10:52:00 -
[151]
Edited by: Risingson on 17/05/2011 10:52:29
i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
most files working ok but i get: MethodCall.server.map.GetDeadspaceAgentsMap.EN ... NO HEADER (<type 'list'>) MethodCall.server.map.GetDeadspaceComplexMap.EN ... NO HEADER (<type 'list'>)
unfortunately thats exactly what i am looking for. help appreciated since i am a py n00b. ty! // Eveeye.com :: IPS Solarsystem Intel & Info |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 11:07:00 -
[152]
Originally by: Risingson i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
I mean the cache file causing the crash, silly... _
Got Item? | EVE API? | Cache? |
Risingson
Mezzanine Inc
|
Posted - 2011.05.17 11:16:00 -
[153]
Originally by: Entity
Originally by: Risingson i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
I mean the cache file causing the crash, silly...
uhmm, confused by your answer... is it somehow possible to get the data out of those 2 cache files? // Eveeye.com :: IPS Solarsystem Intel & Info |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 11:48:00 -
[154]
Originally by: Risingson
Originally by: Entity
Originally by: Risingson i am using this: https://github.com/ntt/reverence/blob/master/examples/datadump.py
I mean the cache file causing the crash, silly...
uhmm, confused by your answer... is it somehow possible to get the data out of those 2 cache files?
Thanks for reminding me why I do not have a customer support job
Originally by: Risingson "narrowed it down to one cache file (d03c.cache - it has the string 'charGetNewTransactions' in the header)"
I want that -file-. Put it somewhere so I can inspect it myself.
_
Got Item? | EVE API? | Cache? |
Risingson
Mezzanine Inc
|
Posted - 2011.05.17 12:07:00 -
[155]
Edited by: Risingson on 17/05/2011 12:09:05
Originally by: Entity Thanks for reminding me why I do not have a customer support job
hehe. sorry should have thought of that by myself.... 86b8.cache and 9554.cache, datadump.py
thank you! // Eveeye.com :: IPS Solarsystem Intel & Info |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.17 12:18:00 -
[156]
Edited by: Entity on 17/05/2011 12:23:48 Edited by: Entity on 17/05/2011 12:19:31
Originally by: Risingson Edited by: Risingson on 17/05/2011 12:09:05
Originally by: Entity Thanks for reminding me why I do not have a customer support job
hehe. sorry should have thought of that by myself.... 86b8.cache and 9554.cache, datadump.py
thank you!
Ah, yeah, they're not database tables, so datadump.py does not dump those. You should write code yourself to deal with these.
do
obj = blue.marshal.Load(file)
then obj[1]['lret'] is a list of tuples it seems. other than the systemID and name I don't know what the 2 other values in there are though.
Also, seems I got confused by your earlier post, sorry for that ^.^ _
Got Item? | EVE API? | Cache? |
Risingson
Mezzanine Inc
|
Posted - 2011.05.17 12:46:00 -
[157]
ok i will try. i've never coded a single line of py though =) thanks for looking into this!
// Eveeye.com :: IPS Solarsystem Intel & Info |
Mike'P
|
Posted - 2011.05.17 19:10:00 -
[158]
Originally by: Entity
Interesting. Can you put this file somewhere I can download it?
Well, I can't give you that particular file as it has now changed and doesn't exhibit the problem. However, I can give you another file which is even more screwy - it reports 4 shared objects, but if you get the old hex dumper out, there is quite clearly not 4 lowball int32_t's at the end of the file.
Grab it here....
http://www.team59.net/66d6.cache
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.18 00:54:00 -
[159]
Originally by: Mike'P
Originally by: Entity
Interesting. Can you put this file somewhere I can download it?
Well, I can't give you that particular file as it has now changed and doesn't exhibit the problem. However, I can give you another file which is even more screwy - it reports 4 shared objects, but if you get the old hex dumper out, there is quite clearly not 4 lowball int32_t's at the end of the file.
Grab it here....
http://www.team59.net/66d6.cache
hm, looking at that file, it shows 2 ints at the end, and the count (the int after the very first byte) also says 2, and it decodes without error here (XP 32bit) I'm confused now :) _
Got Item? | EVE API? | Cache? |
Mike'P
|
Posted - 2011.05.18 12:30:00 -
[160]
Hmm, I just checked that file and you are 100% right, of course! Somehow a running Eve must have overwritten that cache file whilst I was flailing around trying to deduce what was going on.
Anyway, if you try the link again :-
http://www.team59.net/66d6.cache
*that* file has a '04' as the first int after the first byte - and its a GetOrder... head and tail below...
C:\Documents and Settings\Administrator\Desktop\66d6.cache 18,391 bytes 18/05/2011 13:19:42
-0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -A -B -C -D -E -F
00000000- 7E 04 00 00 00 2C 14 04 11 88 13 09 47 65 74 4F [~....,......GetO] 00000001- 72 64 65 72 73 2F 04 AB 96 98 00 05 74 60 16 05 [rders/......t`..] ...... 0000047B- 72 65 74 1B 02 13 07 76 65 72 73 69 6F 6E 2F 08 [ret....version/.] 0000047C- A3 11 DE 1A BD 14 CC 01 13 05 72 75 6E 69 64 02 [..........runid.] 0000047D- 00 00 00 01 00 00 00 [....... ]
I'm on AMD 64 bit, Linux 2.6.32, Python 2.6.2, for the record.
|
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.18 17:17:00 -
[161]
Originally by: Mike'P Anyway, if you try the link again :-
Ok, I've checked that file and did some occult experiments. I can safely say it's either corruption, a failed save, or a bug in EVE. See, the map counter requires that many ints at the end of the file, because there will be that many distinct TYPE_REF tokens in the stream. I think not even EVE itself will know how to decode files with such corrupted maps; there is strong evidence for this in blue.dll in the form of a string saying "Bogus map data in marshal stream".
There is no fix for this, but I will make it detect this case and throw a proper exception. (probably one saying "Bogus map data in marshal stream" )
I'll update Reverence SoonÖ.
Thank you for pointing out this issue :) _
Got Item? | EVE API? | Cache? |
Mike'P
|
Posted - 2011.05.18 20:05:00 -
[162]
Well depressingly, it seems to be happening rather often on my setup!
Admittedly, I'm probably nudging the envelope slightly as this is an Eve running on a Linux laptop using wine with the cache folder mounted via samba on another Linux box which is running the actual Reverence code.
But even so...
The annoyance is that the code I'm trying to use load's the entire cache folder - and one duff cache file causes the whole thing to fall over. I guess if this particular case is trapped properly I can manage that exception - thanks for looking into it.
I think I reported it more to shine a light. rather than expecting any direct help!
So thanks muchly - oh, and great library, BTW ;-)
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.05.25 17:04:00 -
[163]
1.2.2 released. The only change is the fix for the corrupt shared object map issue above, but updating doesn't hurt :) _
Got Item? | EVE API? | Cache? |
Guido Calabresi
|
Posted - 2011.06.10 02:57:00 -
[164]
I can confirm that the datadump.py script runs properly on Mac OS X with the (more or less) default python install if you manually configure the path to the EVE directory inside the application package.
|
Sullinae
|
Posted - 2011.06.23 20:44:00 -
[165]
busted for Incarna, needs to be updated for the new cache formats please update this great tool, I want to start making piles of isk with my trade finding script again!
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.23 22:16:00 -
[166]
Originally by: Sullinae busted for Incarna, needs to be updated for the new cache formats please update this great tool, I want to start making piles of isk with my trade finding script again!
It'd be nice if you could be a tad more specific? _
Got Item? | EVE API? | Cache? |
Sullinae
|
Posted - 2011.06.25 04:43:00 -
[167]
Edited by: Sullinae on 25/06/2011 04:43:49 Ah, nevermind. Thought it had something to do with them changing the order ID from 16 to 32 bit ints.
It was just that they took something out of the DBRows... or renamed it:
Getting an error: AttributeError: 'blue.DBRow' object has no attribute 'issued'
Seems to work other than that now.
Thanks for all your hard work btw, this library rocks my socks.
|
locJ
|
Posted - 2011.06.25 08:05:00 -
[168]
Edited by: locJ on 25/06/2011 08:08:02 I Found a Problem Some market items are ignored. Specifically some subsystems, such as Legion defensive subsystems.
Im using Reverence 1.2.2 with an autodumper Script for Eve market Scanner (ems)
Other then this, gerat stuff. Keep up the good work
|
Kandreath
|
Posted - 2011.06.25 11:01:00 -
[169]
I've worked through a couple of issues but this one has me stumped. I can't see it covered here at all.
Using Python 2.7 (amd64) on Windows 7 (64bit)
I'm trying to compile from source as instructed for a win7-64 setup. I installed vc 2010 and couldn't get it to work. I then un-installed and found vc 2008 and installed that. That got me over the first error.
(I'm including the story in case it's significant).
C:\Users\[...]\Downloads\reverence-1.2.2>C:\Python27\python.exe setup.py install running install running build running build_py creating build\lib.win-amd64-2.7 creating build\lib.win-amd64-2.7\reverence copying src\blue.py -> build\lib.win-amd64-2.7\reverence copying src\cache.py -> build\lib.win-amd64-2.7\reverence copying src\config.py -> build\lib.win-amd64-2.7\reverence copying src\const.py -> build\lib.win-amd64-2.7\reverence copying src\dbutil.py -> build\lib.win-amd64-2.7\reverence copying src\embedfs.py -> build\lib.win-amd64-2.7\reverence copying src\exceptions.py -> build\lib.win-amd64-2.7\reverence copying src\objectCaching.py -> build\lib.win-amd64-2.7\reverence copying src\strings.py -> build\lib.win-amd64-2.7\reverence copying src\util.py -> build\lib.win-amd64-2.7\reverence copying src\_os.py -> build\lib.win-amd64-2.7\reverence copying src\__init__.py -> build\lib.win-amd64-2.7\reverence running build_ext building 'reverence._blue' extension Traceback (most recent call last): File "setup.py", line 63, in <module> package_data = {"reverence": ['*.txt']}, File "C:\Python27\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands self.run_command(cmd) File "C:\Python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Python27\lib\distutils\command\install.py", line 563, in run self.run_command('build') File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\Python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Python27\lib\distutils\command\build.py", line 127, in run self.run_command(cmd_name) File "C:\Python27\lib\distutils\cmd.py", line 326, in run_command self.distribution.run_command(command) File "C:\Python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run() File "C:\Python27\lib\distutils\command\build_ext.py", line 340, in run self.build_extensions() File "C:\Python27\lib\distutils\command\build_ext.py", line 449, in build_exte nsions self.build_extension(ext) File "C:\Python27\lib\distutils\command\build_ext.py", line 499, in build_exte nsion depends=ext.depends) File "C:\Python27\lib\distutils\msvc9compiler.py", line 473, in compile self.initialize() File "C:\Python27\lib\distutils\msvc9compiler.py", line 383, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal l raise ValueError(str(list(result.keys()))) ValueError: [u'path']
Any Idea?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.25 12:52:00 -
[170]
Originally by: Kandreath
File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal l raise ValueError(str(list(result.keys()))) ValueError: [u'path']
Any Idea?
Hmmm, I looked around a bit, and only found: http://stackoverflow.com/questions/4676728/value-error-trying-to-install-python-for-windows-extensions
Quote: If you have a 64 bit Python installation: Install "Microsoft Visual Studio 2008 Professional Edition" with the "X64 Compiler and Tools" option enabled.
I cannot test this myself yet as i'm using 32bit windows and too lazy to update. (I did buy a new laptop this week, however it's DOA, so so much for that idea)
_
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.06.25 12:57:00 -
[171]
Originally by: Sullinae Getting an error: AttributeError: 'blue.DBRow' object has no attribute 'issued'
Seems to work other than that now.
Keep in mind CCP can and does change the structure and contents of cache files whenever they feel like. Errors like these are nearly always indicative of such a change.
In this case, CCP simply renamed the "issued" column to "issueDate". Quite far from 'busted for incarna' ;)
Originally by: Sullinae Thanks for all your hard work btw, this library rocks my socks./quote]
Cheers :) _
Got Item? | EVE API? | Cache? |
Kandreath
|
Posted - 2011.06.25 22:14:00 -
[172]
Originally by: Entity
Originally by: Kandreath
File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsal l raise ValueError(str(list(result.keys()))) ValueError: [u'path']
Any Idea?
Hmmm, I looked around a bit, and only found: http://stackoverflow.com/questions/4676728/value-error-trying-to-install-python-for-windows-extensions
Quote: If you have a 64 bit Python installation: Install "Microsoft Visual Studio 2008 Professional Edition" with the "X64 Compiler and Tools" option enabled.
I cannot test this myself yet as i'm using 32bit windows and too lazy to update. (I did buy a new laptop this week, however it's DOA, so so much for that idea)
I tried to use the python uitls offered as an alternative in the stack overflow reply, but this didn't help. Decided to use the 32bit Python 2.7 install as the 2008 professional edition has only a 90 day trial and I recon I'll want to compile this again after the 90 days. This seemed to do the trick and I'm up and running now.
Thanks for the quick reply it solved my problem.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.07.27 01:35:00 -
[173]
I've just committed a few crash bug fixes. The fixes are for a reference count bug in virtual column names, and directly nested arbitrary-size containers being prematurely cut because of not clearing the TYPE_MARK state.
These bugs don't affect most (if any at all) users, so no new installers yet.
Note that Reverence is currently incompatible with the build on Singularity due to CCP deploying a shiny new bulkdata management system. The necessary changes have been done to make Reverence work with that, but I will not push these changes until TQ is updated, as things may still change.
_
Got Item? | EVE API? | Cache? |
Fena Echerie
|
Posted - 2011.07.29 10:28:00 -
[174]
Hi,
I just downloaded your lib and tried datadump.py. I see lots of xml files (mostly bulk data) but the market data I want is missing. Snip of market data related errors: MethodCall.proxy.marketProxy.GetRegionBest.10000016L ... UNSUPPORTED (dbutil.RowDict) MethodCall.proxy.marketProxy.GetMarketGroups ... FAILED: Unsupported type // I edited datadump.py to report error type
When looking for a solution I found your last post, saying the windows installers are out of date, so I tried to compile the sources myself. However: > python setup.py build <snip> building 'reverence._blue' extension error: Unable to find vcvarsall.bat
Since I have Visual Studio 10 Professional installed and already tried adding the location of vcvarsall.bat to the PATH variable, I'm kind of lost now. Any help is appreciated :)
btw: Great work! Reverence makes a much better and cleaner impression than libevecache, which is passing/returning std::vectors by value all the time... |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.07.29 13:16:00 -
[175]
Originally by: Fena Echerie Hi,
I just downloaded your lib and tried datadump.py. I see lots of xml files (mostly bulk data) but the market data I want is missing. Snip of market data related errors: MethodCall.proxy.marketProxy.GetRegionBest.10000016L ... UNSUPPORTED (dbutil.RowDict) MethodCall.proxy.marketProxy.GetMarketGroups ... FAILED: Unsupported type // I edited datadump.py to report error type
When looking for a solution I found your last post, saying the windows installers are out of date, so I tried to compile the sources myself. However: > python setup.py build <snip> building 'reverence._blue' extension error: Unable to find vcvarsall.bat
Since I have Visual Studio 10 Professional installed and already tried adding the location of vcvarsall.bat to the PATH variable, I'm kind of lost now. Any help is appreciated :)
btw: Great work! Reverence makes a much better and cleaner impression than libevecache, which is passing/returning std::vectors by value all the time...
The dump script was made for bulkdata, not for dynamic stuff like market data. If you want to process market data you will have to write your own code for it (there's several examples in this thread iirc)
Also, to compile it you need VS9 I think. Python's build scripts are picky :)
_
Got Item? | EVE API? | Cache? |
CheckingAmarr
|
Posted - 2011.08.12 03:25:00 -
[176]
I have an old script which looks at market orders and then does some calculations with item blueprints, specifically through looking up cfg.ramtypematerials values. I noticed on page 4 there was some discussion of how ccp had broken that, but was there any real resolution of the issue? How do I access blueprint info for items ingame now? I am terrible at eve and python and not sure of this, and my script is just erroring out with a "cache file not found: 9ba4.cache" error.
I realize this isn't a tech support thread but any comments or examples would be greatly appreciated. Thanks.
|
Theodor Bliesheimer
Integrated Astromechanics Corp. Zeta Conglomerate
|
Posted - 2011.08.24 15:19:00 -
[177]
Originally by: Entity
Note that Reverence is currently incompatible with the build on Singularity due to CCP deploying a shiny new bulkdata management system. The necessary changes have been done to make Reverence work with that, but I will not push these changes until TQ is updated, as things may still change.
Could CCP have already released the change you mentioned? Since about two days ago my market script fails here:
if "GetOldPriceHistory" in key or "GetOrders" in key ... ...
|
Uberfrau
|
Posted - 2011.08.24 19:56:00 -
[178]
Originally by: Guido Calabresi I can confirm that the datadump.py script runs properly on Mac OS X with the (more or less) default python install if you manually configure the path to the EVE directory inside the application package.
Can you tell me what you did to make this work? I've tried these:
/Applications/EVE\ Online.app/Contents/Resources/transgaming/c_drive/Program\ Files/CCP/EVE/ /Applications/EVE\ Online.app/Contents/Resources/transgaming/c_drive/Program\ Files/CCP/EVE
I tried symlinking eve in the root (/eve) to see if it was an issue with spaces in the path, but I still get "Could not determine EVE cache folder location."
This is running Incursion on Lion, 10.7.1.
|
Uberfrau
|
Posted - 2011.08.24 21:19:00 -
[179]
Originally by: Manathera Edited by: Manathera on 18/12/2010 21:13:13 I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable. MarketReader Please let me know of any errors as this is the first python code I've ever written.
I get a Segmentation fault: 11 :]
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.08.24 22:15:00 -
[180]
Originally by: Uberfrau
Originally by: Manathera Edited by: Manathera on 18/12/2010 21:13:13 I just discovered reverence and love it, thank you entity. So I thought I'd contribute to this by providing a bit of code I wrote. based on a sample posted by Cyaxares II on the first page. His code just gets the high, low and average price history. My sample gets the entire order and outputs it to a file that's as identical to the market export files as possable. MarketReader Please let me know of any errors as this is the first python code I've ever written.
I get a Segmentation fault: 11 :]
Quite possibly an issue that's been resolved already. If you can, checkout the repository and build from that. otherwise just wait for the new release (which should be when CCP switches over to the new bulkdata system) _
Got Item? | EVE API? | Cache? |
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.01 22:04:00 -
[181]
Reverence 1.3.0 now available (including installers). This release is compatible with the new Incarna 1.1 bulkdata system (but is backwards compatible).
_
Got Item? | EVE API? | Cache? |
MFG Overload
Minmatar Republic University
|
Posted - 2011.09.02 08:46:00 -
[182]
After update I can't get "evelocations" work. cfg.evelocations.Get(key[2]) always print DBRow(locationID:0,locationName:None,x:0.0,y:0.0,z:0.0)
Thank you :)
|
Theodor Bliesheimer
Integrated Astromechanics Corp. Zeta Conglomerate
|
Posted - 2011.09.02 08:52:00 -
[183]
Edited by: Theodor Bliesheimer on 02/09/2011 08:53:16 Awesome work, thanks!
I've encountered a small problem with the new version though. Without going into intricate details, are you aware of any changes to the getconfigmgr.evelocations class, especially to locationName?
EDIT: Haha, was typing this while the guy above made his post.
|
Aineko Macx
|
Posted - 2011.09.02 09:46:00 -
[184]
Have you ever considered rewriting blue in python to remove the dependency on a compiled C extension?
Keep up the good work! ________________________ CCP: Where fixing bugs is a luxury, not an obligation. |
Deerin
Minmatar Murientor Tribe
|
Posted - 2011.09.02 15:05:00 -
[185]
Edited by: Deerin on 02/09/2011 15:05:21 Thank you for the fast update.
Came here to make a little bug report about cfg.evelocations.get() But appearently I wasn't alone. It returns a DBRow(locationID:0,locationName:None,x:0.0,y:0.0,z:0.0). Also crashes python in some cases ------------------------------------------- Die Amarr Die!!! |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 15:46:00 -
[186]
yeah evelocations and eveowners are broken because I messed up with the DBRow initialization <_<. Fix SoonÖ _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 17:37:00 -
[187]
Edited by: Entity on 02/09/2011 17:37:13 Version 1.3.1 now available.
This hopefully fixes the eveowners and evelocations tables (I tested it though, so it should work fine ;)
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.02 18:38:00 -
[188]
Originally by: Aineko Macx Have you ever considered rewriting blue in python to remove the dependency on a compiled C extension?
Reverence actually used to be pure python before I released it. It works, but it's terribly slow and uses a huge amount of memory to load tables.
_
Got Item? | EVE API? | Cache? |
Deerin
Minmatar Murientor Tribe
|
Posted - 2011.09.02 21:56:00 -
[189]
Thank you very much for this fast fix. It is working flawlessly again. ------------------------------------------- Die Amarr Die!!! |
tantomonta montatanto
|
Posted - 2011.09.02 23:03:00 -
[190]
Originally by: Entity Edited by: Entity on 02/09/2011 17:37:13 Version 1.3.1 now available.
This hopefully fixes the eveowners and evelocations tables (I tested it though, so it should work fine ;)
Thank you!
|
|
Jaff Rockline
|
Posted - 2011.09.03 20:09:00 -
[191]
Great library. I've been lurking around here for awhile and using the library with autodumper to great success - beautiful work Entity! Unfortunately, that all came crashing down with Incarna 1.1. I started getting errors as most people did, so I updated Reverence to 1.3.1 using 'reverence.1.3.1.win32-py2.6.exe'. The errors continued, so I grabbed the source, ran a build and installed. Still the same errors. When, in game, a market item in accessed, the following rolls in the python window:
error wiring data: 'blue.DBrow' object has no attribute 'name'
I understand from reading the entire thread many times that these attribute names can change - I'm surprised no one else has mentioned this, unless it is just me.
So - my question is twofold. 1. What must I do to fix this issue? 2. Is there a way I can list the current object attributes myself from the files so I can make the needed edits on my own?
Thanks for the help - and again - great work!
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.03 20:31:00 -
[192]
Originally by: Jaff Rockline
So - my question is twofold. 1. What must I do to fix this issue? 2. Is there a way I can list the current object attributes myself from the files so I can make the needed edits on my own?
Thanks for the help - and again - great work!
Indeed, attribute errors are pretty much always changes by CCP. To see the attributes on a row, just print the row itself to console. It should give some output like this:
DBRow(someAttribute:1,someOtherAttribute:2,...bladiblabla)
and you should be able to see what to do from there.
_
Got Item? | EVE API? | Cache? |
Dr Bernard
|
Posted - 2011.09.03 20:59:00 -
[193]
Originally by: Jaff Rockline
What must I do to fix this issue?
In your main.py (assuming you are using the autodumper scripts), change region = self.eveConfig.evelocations.Get(region_id).name to region = self.eveConfig.evelocations.Get(region_id).locationName
I don't know anything either about debugging this kind of thing as I'm a about as useful as my gran when it comes to programming skills.
If anyone can give some hints or tips or recommend a guide on figuring this sort of voodoo out I would also be eternally greatful.
Again great work Entity.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.03 21:15:00 -
[194]
Edited by: Entity on 03/09/2011 21:26:47
Hm, ah, I see what the issue is. It's due to the way those tables are made now. Not entirely sure what to do about this. I'll look into it.
Right. It's actually a tiny oversight on my part. See, I use a slightly different method for loading tables than EVE does, which is ever so slightly faster and adds more features and flexibility to the datasets, but for a few of those recently changed tables I used the normal (compatible) way, which is more strict about things (and possibly bugged, but nobody should be using compatibility mode anyway).
Fix SoonÖ _
Got Item? | EVE API? | Cache? |
Jaff Rockline
|
Posted - 2011.09.04 03:43:00 -
[195]
Originally by: Entity
Originally by: Jaff Rockline
So - my question is twofold. 1. What must I do to fix this issue? 2. Is there a way I can list the current object attributes myself from the files so I can make the needed edits on my own?
Thanks for the help - and again - great work!
Indeed, attribute errors are pretty much always changes by CCP. To see the attributes on a row, just print the row itself to console. It should give some output like this:
DBRow(someAttribute:1,someOtherAttribute:2,...bladiblabla)
and you should be able to see what to do from there.
Thanks so much for the reply. I kinda thought it was something like that. I guess my problem is that I am not certain how to read those rows, and which files to read. Is it something like: dump = (pathtofile) f = open(dump, 'rb') filedump = f.read() f.close obj = blue.marshal.Load(filedump) print obj
If I do that, I get a lot of unusable stuff. Sorry for the elementary questions - I'm trying to get myself into this a little deeper.
Thanks again |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.04 15:59:00 -
[196]
1.3.2 available.
lets you use the "name" attribute alias for eveowners and evelocations again :) _
Got Item? | EVE API? | Cache? |
Dogius Meatius
|
Posted - 2011.09.05 15:55:00 -
[197]
Hoping someone can help. Autodumper scripts no longer work with latest Eve patch and this new Reverence. I am using Pytho 2.7, pywin32-214, new reverence from today. Autodumper runs but just shows black screen/no text. I tried clearing out cache and so on, no dice!
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2011.09.08 18:18:00 -
[198]
New forums thread: https://forums.eveonline.com/default.aspx?g=posts&t=6501 _
Got Item? | EVE API? | Cache? |
|
|
|
Pages: 1 2 3 4 5 6 7 :: [one page] |