Pages: 1 [2] 3 4 5 6 7 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.05.15 13:58:00 -
[31]
Edited by: Entity on 15/05/2010 13:59:03
Originally by: cinderbrood Edited by: cinderbrood on 15/05/2010 04:38:11 Will we ever see a pure python version of this without the c/c++ extensions?
(I want to run this on IronPython) but alas they lack .pyd support.
Ie will we ever see a pure python version of your blue.pyd?
would open up cache support to the .net world as well. (albeit in a hacky manner)
Heh. It actually used to be pure Python, it was mostly a pet research project. Worked fine, and I've used it like that for a long time, but there's a couple of problems with it:
- I'd end up having to maintain the same thing in two different languages.
- Memory usage goes through the roof because of Python data structure overhead.
- Without acceleration (Psyco jit compiler) it is painfully slow.
_
Got Item? | EVE API? | Cache? |
cinderbrood
Caldari Delta Technologies And Development
|
Posted - 2010.05.15 20:40:00 -
[32]
Aww Damn it :P Speed wasnt an issue for me either ( Within reason i guess ) As for memory usage.. Yeah thats not going to get any better with IronPython i guess. Any chance of the old release being archived somewhere for those of us with pet projects themselfs that want to hack at it :P (Short of writing an entire .net version )
Writing my own cache reading libs was something im trying to avoid as its just an addon. not a required part of my little play toy.
Will Code For Isk. |
Miss AmarrPriceCheck
|
Posted - 2010.05.26 21:42:00 -
[33]
On a previously working script, I now have the following error post-Tyrannis:
Quote:
Traceback (most recent call last): File "C:\Documents and Settings\User\Desktop\scriptname.py", line 54, in <module> if str(cfg.evelocations.Get(key[2]).locationName) == "Lonetrek": File "C:\Python26\lib\site-packages\reverence\config.py", line 338, in __get__ value = self.method(obj) File "C:\Python26\lib\site-packages\reverence\config.py", line 606, in evelocations self._loadfrombulk("config.StaticLocations", rs, hint=1) File "C:\Python26\lib\site-packages\reverence\config.py", line 734, in _loadfrombulk rs.lines.extend(obj.lines) AttributeError: 'CRowset' object has no attribute 'lines'
Sorry the whitespace formatting in quoted error is obviously wrong, but you get the idea.
|
Kadesh Priestess
Scalding Chill
|
Posted - 2010.05.26 22:27:00 -
[34]
Entity updated reverence to work with tyrannis - you need to get and install latest version from git or if you prefer installer - wait for an update. Git version works perfectly for me.
|
Jita Grrl
|
Posted - 2010.06.13 12:06:00 -
[35]
Originally by: Entity Edited by: Entity on 03/10/2009 02:09:45 100% compatibility with all bulkdata, cache and settings files.
What exactly do you mean when you said settings files?
I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.
Is this something Reverence can do or am I barking up the wrong tree?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.13 19:19:00 -
[36]
Originally by: Jita Grrl
Originally by: Entity Edited by: Entity on 03/10/2009 02:09:45 100% compatibility with all bulkdata, cache and settings files.
What exactly do you mean when you said settings files?
I'm mainly looking to pull window sizes and positions from the cache. It looks like that data should be stored in the various core_char_XXXXXXXX.dat files, but I'm not certain.
Is this something Reverence can do or am I barking up the wrong tree?
Yeah it can do that, simply do something like
obj = blue.marshal.Load(contents_of_settings_file)
settings files are mostly just dictionary mappings.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.06.17 13:35:00 -
[37]
Edited by: Entity on 17/06/2010 13:35:44 A windows installer and source distribution for version 1.1.0 are now available (see OP for location) for your convenience.
Version 1.1.0 fixes various bugs and other issues and is compatible with Tyrannis, and adds a couple of features. The most useful change being that DBRows now have a string representation, which is handy for debugging and checking out table contents.
Check the commit history on github for more details.
Note: Users already using the current version directly from the repository don't have to do anything.
_
Got Item? | EVE API? | Cache? |
Veevi Fenikusu
Minmatar Wan Futuristics
|
Posted - 2010.07.09 11:38:00 -
[38]
This library is truly amazing, great work!
|
Veevi Fenikusu
Minmatar Fenikusu Trade
|
Posted - 2010.07.17 14:44:00 -
[39]
Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.17 15:47:00 -
[40]
Originally by: Veevi Fenikusu Do have a question about using this! If you want to use this to monitor checking market orders, should you first browse everything you want to import, then run the cachemgr stuff, run it each time the cache folder changes, or run it at certain intervals(I'm doing it like this now, but it got to run through the entire cache everytime, and it gets bigger...)
I would just monitor the folder for new files and only check those after examining the existing ones. Also, you can decode the files manually with blue.marshal.Load(data) instead of using cachemgr. A further optimization would be to just inspect the binary data and see whether it's a market order before loading/decoding it. _
Got Item? | EVE API? | Cache? |
|
Gogo cyno
|
Posted - 2010.07.19 16:45:00 -
[41]
I was playing around with this and had a question about the cachemgr's GetCacheFileName method:
Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.07.20 12:06:00 -
[42]
Originally by: Gogo cyno I was playing around with this and had a question about the cachemgr's GetCacheFileName method:
Through time stamps and various other things, I know that a file (say 6047.cache) is the file for corp orders (key ('marketProxy', 'GetCorporationOrders', 1234) after loading the folder). How do I call GetCacheFileName (or thus any of the load cached file ones) to access that file?
GetCacheFileName is kind of for internal usage and manual loading (by loading the file and decoding with blue.marshal.Load). If you know the cache key of a file (the tuple you mentioned), you can do this if the file is in CachedMethodCalls:
obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )
and this if it's in CachedObjects:
obj = cache.LoadCachedObject( ('marketProxy', 'GetCorporationOrders', 1234) )
Note that LoadCacheFolder is a one-shot thing and it actually loads all the data at once, so it is very inefficient if you only use one object. The method is provided mostly for debugging/dumping.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.07.20 15:29:00 -
[43]
Edited by: Gogo cyno on 20/07/2010 15:36:10
Originally by: Entity
obj = cache.LoadCachedMethodCall( ('marketProxy', 'GetCorporationOrders', 1234) )
Alas, I was hoping I was just doing it wrong. Whenever I try to load a specific method call in the above manner, I get a No Such File or Directory error and the filename will be different than what I know the cache file is. For example,
r = cachemgr.LoadCachedMethodCall( ('marketProxy', 'GetOrders', 10000002, 40) )
will produce that error. I know that method call is there because I can access it through loading the entire folder, but trying to load it specifically does not work.
cmc[('marketProxy', 'GetOrders', 10000002, 40)]
will return the correct information (with cmc = cachemgr.LoadCacheFolder("CachedMethodCalls") ) |
Saphira Khan
The Wretched.
|
Posted - 2010.07.30 16:44:00 -
[44]
I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them. www.eve-lotteries.com |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 18:52:00 -
[45]
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
_
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.02 20:04:00 -
[46]
Originally by: Entity
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
Hm it looks like the CachedMethodCalls folder uses a different hash key than the CachedObjects one which works perfectly. I'll look into it.
Hm, it looks like it's actually a decoding problem, but it's so incredibly subtle I don't understand why it's failing.
The problem: cPickle produces a different pickle string for two seemingly identical tuples.
Example: cachedmethodcall for t2 expanders in Forge:
('marketProxy', 'GetOrders', 10000002, 1319)
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
The only difference here are the objectIDs differing. Even comparing the key to the constant ('marketProxy', 'GetOrders', 10000002, 1319) produces True.
This shouldn't really matter, but apparently it does.
A workaround is to re-tuple the key by doing key = key[:] before using it.
At the moment I have no idea why it behaves like that, and I'm tempted to use that workaround in the library.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.08.02 20:38:00 -
[47]
Originally by: Entity
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
>> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) 'd58e.cache'
And yet another result. I tried using the workaround you mentioned with no success.
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 00:56:00 -
[48]
Originally by: Gogo cyno
Originally by: Entity
GetCacheFileName produces "986f.cache" for the key pulled from the cache file, but if you do GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) manually, it produces the correct "f42f.cache".
>> cachemgr.GetCacheFileName(('marketProxy', 'GetOrders', 10000002, 1319)) 'd58e.cache'
And yet another result. I tried using the workaround you mentioned with no success.
I get the distinct feeling cPickle has some undocumented behaviour.
what does cPickle.dumps(('marketProxy', 'GetOrders', 10000002, 1319)) give? _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 01:35:00 -
[49]
oh wow. Found the cause of the problem. cPickle is not stable for given input depending on refcounts.
Originally by: picklebug.py
import cPickle
reffed = "xKITTENSx"[1:-1] print repr(cPickle.dumps(reffed))
print repr(cPickle.dumps("xKITTENSx"[1:-1]))
Produces:
F:\>picklebug.py "S'KITTENS'\np1\n." "S'KITTENS'\n."
Note the extra "p1\n" in the 1st case.
Very strange... Not sure how to solve this. seems to be a cPickle issue.
More worryingly, since the method I use to generate these cache filenames is exactly identical to what EVE does, I wonder whether EVE would have the same problem in some cases.
_
Got Item? | EVE API? | Cache? |
Gogo cyno
|
Posted - 2010.08.03 14:30:00 -
[50]
Using pickle seems to produce consistent results:
import pickle
reffed = "xKITTENSx"[1:-1] print repr(pickle.dumps(reffed))
print repr(pickle.dumps("xKITTENSx"[1:-1]))
produces:
"S'KITTENS'\np0\n." "S'KITTENS'\np0\n."
|
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 15:26:00 -
[51]
Yep, but not identical to cPickle. Consistency doesn't help if the string is still different, as the hash code will be different too. _
Got Item? | EVE API? | Cache? |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:09:00 -
[52]
Update
I've found evidence of EVE also being affected by this, as I have found cache entries for calls to the same method, with different arguments, that exhibit the same pickling inconsistency:
"(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVX-Factor Ind\nI0\ntp3\n." <- only three PUTs "(S'lookupSvc'\np1\nS'LookupOwners'\np2\nVNTT\np3\nI0\ntp4\n." <- four PUTs
So, it's pretty much impossible to know whether EVE held more than 1 reference to a string when it hashed the cache key, and as such there's no good way to solve this problem.
Recursively collecting all elements of the key into a new list so that their reference count is going to be at least 2 before using GetCacheFileName() does seem to fix it for most of the cache entries, but there's still going to be a handful that fail to produce the correct hash.
Try the following workaround, but note that you should only do this for the CachedMethodCalls folder.
def collect(key): foo = [] _collect((key,), foo.append) return foo
def _collect(items, append): for element in items: if type(element) in (list, tuple): _collect(element, append) append(element)
foo = collect(key) filename = cm.GetCacheFileName(key)
Guess I'll be poking some people in CCP about this.
_
Got Item? | EVE API? | Cache? |
Catari Taga
Centre Of Attention Middle of Nowhere
|
Posted - 2010.08.03 18:22:00 -
[53]
Originally by: Entity Guess I'll be poking some people in CCP about this.
Please follow the normal procedure and explain the bug to a bughunter first. --
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 18:55:00 -
[54]
Originally by: Catari Taga
Originally by: Entity Guess I'll be poking some people in CCP about this.
Please follow the normal procedure and explain the bug to a bughunter first.
_
Got Item? | EVE API? | Cache? |
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2010.08.03 20:22:00 -
[55]
Pam-pam-pam... This could likely be the case for "stuck" corporation assets listings. -- Thanks CCP for cu |
SmallBrain
|
Posted - 2010.08.03 21:58:00 -
[56]
Originally by: Saphira Khan I might have a simular problem. When i do the following: cmc = LoadCacheFolder("CachedMethodCalls") for key, obj in cmc.iteritems(): filename = cachemgr.GetCacheFileName(key)
it returns a filename but this file doesnt exist anywhere in my cache folders. Any idea how i can solve this? I ideally want to remove cache files after i used them so i wont have to parse / send them again after i've already processed them.
A bug still being a bug, the filename of course should exist, but I solved this differently. This snippet might help other people too:
def LoadOurCacheFolder(cache, name): crap = {} if cache.machoVersion > 181 and name.lower() == "bulkdata": name = cache.bulkdatapath else: name = os.path.join(cache.machocachepath, name) for filename in glob.glob(os.path.join(name, "*.cache")): try: what, obj = blue.marshal.Load(open(filename, "rb").read()) except IOError: continue obj['filename'] = filename crap[what] = obj return crap
This assigns the filename to the object too. Then I just do:
cmc = LoadOurCacheFolder(cache, "CachedMethodCalls") for key, obj in cmc.iteritems(): os.unlink(obj['filename'])
Well .. the unlink is done when I know for sure the data is posted, but you get the idea :) So now the globl always returns new files. And a nice side-effect: it keeps the cache-dirs relative empty, and keep it from consuming tons of MiB when you scan the complete market in a few regions ;)
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.03 22:21:00 -
[57]
yep, that works too.
I've send off some mails to the right people so hopefully the bug is going to be fixed sometime SoonÖ and i'll update Reverence :)
_
Got Item? | EVE API? | Cache? |
Saphira Khan
The Wretched.
|
Posted - 2010.08.11 17:56:00 -
[58]
Thanks for the help all, will try this soon! www.eve-lotteries.com |
Induc
Amarr
|
Posted - 2010.08.17 20:10:00 -
[59]
Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?
|
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2010.08.18 12:32:00 -
[60]
Originally by: Induc Now with Python 2.7 released as the latest stable version, is it possible if Reverence could be updated to support it?
Working on it. It compiles for 2.7 it seems. Will have to check EVE first once it's up, to see if the marshal format and cache key hashing are still the same.
Update SoonÖ _
Got Item? | EVE API? | Cache? |
|
|
|
|
Pages: 1 [2] 3 4 5 6 7 :: one page |
First page | Previous page | Next page | Last page |