| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
roigon
Per.ly
The 20 Minuters
2
   |
Posted - 2011.12.21 09:58:00 -
[1] - Quote
I'm building an application that uses reverence to passively process marketdata. I'm running into an issue though, after a while revenance opens too many files and it throws an exception. (see below)
Is there some cleanup method somewhere I can call for reverence to cleanup it's open files?
Quote:Exception in thread Thread-16:
Traceback (most recent call last):
File "C:\Python27_32b\lib\threading.py", line 552, in __bootstrap_inner
File "C:\Users\erik\git\marketTool\marketTool\src\eve\marketCacheImporter.py", line 42, in run
File "C:\Users\erik\git\marketTool\marketTool\src\eve\eveLib.py", line 18, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\blue.py", line 126, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\blue.py", line 79, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\embedfs.py", line 86, in __init__
File "C:\Python27_32b\lib\site-packages\reverence\embedfs.py", line 32, in __init__
IOError: [Errno 24] Too many open files: 'C:\\Program Files (x86)\\CCP\\EVE2\\resCharacterTattoos.stuff' |
roigon
Per.ly
The 20 Minuters
2
|
Posted - 2011.12.21 19:32:00 -
[2] - Quote
Entity wrote:roigon wrote:IOError: [Errno 24] Too many open files: 'C:\\Program Files (x86)\\CCP\\EVE2\\resCharacterTattoos.stuff'
-edit-
Seem to have fixed the problem or at least made it less likely to happen by explicitly calling the garbage collector after each scan. Looping over a large amount of files without calling close() after each tends to cause this issue. The garbage collector simply kicks in too late for it to stop you running out of filehandles that way. Just explicitly close a file before moving on to the next one and you will have no issue. Also, avoid using LoadCacheFolder() for this sort of thing (if you're using it), I kinda regret putting it in the published version.
That's exactly what I used :)
I kinda liked that reverance did all the I/O for me so I could just treat it like a magic black box that spitted out good stuff, but I guess I'll have to get my hands dirty and actually tell it which cache files to look at and make sure they get processed in smaller batches so i can properly close the files.
Many thanks for creating it though, other then this issue it's been very easy to work with. I wish some of the code had some more comments* but that's also my lack of python knowledge at work.
* Yes I know the answer to this is "patches welcome" :P |
roigon
Per.ly
The 20 Minuters
5
|
Posted - 2012.01.22 13:51:00 -
[3] - Quote
I use the following javascript to loop over items.
http://paste2.org/p/1879109
Please note that [PLACEHOLDER] get's replaced by python with a list of items I want to loop over, and there's some js in there to create a progress bar, probably has a bug though.
But more or less I just queue up the calls with setTimeout with a 3s interval. Works for me. |
roigon
Per.ly
The 20 Minuters
5
|
Posted - 2012.01.24 11:57:00 -
[4] - Quote
I'm trying to move away from using a the database dump and getting more stuff via reverence. However I'm hitting a wall on this one.
What I want to do is build a tree of the market groups. This for configuration purposes of scanning market items. I can easily enough get this from the database but I'm getting a bit stumped when trying to get this info from reverence.
I've tried various of the bulkdata tables from reverence trying to find the info i need. invgroups, groupsByCategories and typesByMarketGroups. However because of my limited python knowledge I have some trouble interpreting the data I'm getting back and handling it.
I'm trying to parse the data in the following way.
http://paste2.org/p/1881080
This however just ends up running out of memory with a seemingly endless amount of records and groups.
If someone could point me into the right direction I would be quite thankful. |
| |
|