| Pages: [1] :: one page |
|
|
| Author |
Thread Statistics | Show CCP posts - 3 post(s) |
postal dude
Thundercats
RAZOR Alliance
   |
Posted - 2008.03.25 22:19:00 -
[1]
As I understand there is no standings export via the API or the eve-online website.
I poked around in the eve-cache folders and stumbled over a file with the corp standings in it. Before I start to implement an app in C# to parse the file: Is it legal or an eula-violation? Has somebody already wrote such an app?
|
|
CCP Lingorm
C C P
|
Posted - 2008.03.26 08:51:00 -
[2]
As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.
If you do try to change it it will not actually effect the corp standings only the data YOU see in your client (which could cause problems for you).
In short : Yes you can do it, please don't change the file.
CCP Lingorm
CCP Quality Assurance
QA Engineering Team Leader
|
|
postal dude
Thundercats
RAZOR Alliance
|
Posted - 2008.03.26 10:10:00 -
[3]
Edited by: postal dude on 26/03/2008 10:11:16
 Originally by: CCP Lingorm
In short : Yes you can do it, please don't change the file.
Thank you very much! It was easy to implement the file-format. The only problem is, that only the standings from your corp are cached, but thats good enough for me:
If anybody wants the source, Iam willing to upload it 
|
|
CCP Lingorm
C C P
|
Posted - 2008.03.26 10:28:00 -
[4]
I wouldn't mind a copy.
CCP Lingorm
CCP Quality Assurance
QA Engineering Team Leader
|
|
Salvis Tallan
The Shadow Order
SMASH Alliance
|
Posted - 2008.03.26 11:27:00 -
[5]
Originally by: CCP Lingorm
I wouldn't mind a copy.
QFT. I like that and would love a copy of the code.
------
|
Ix Forres
Vanguard Frontiers
Imperial Republic Of the North
|
Posted - 2008.03.26 11:57:00 -
[6]
Originally by: CCP Lingorm
As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.
If you do try to change it it will not actually effect the corp standings only the data YOU see in your client (which could cause problems for you).
In short : Yes you can do it, please don't change the file.
Does this apply to all cache data? I'm thinking market data, if that's cached when browsed. Would make for a much more effective market report uploader for a site I'm working on... 
Blog |
Pronas
Scoopex
The Threshold
|
Posted - 2008.03.26 11:57:00 -
[7]
nice tools
would like a copy too (as well as source ;)
--
The Threshold Diplomat
Scoopex CEO
Scoopex is Recruting |
Vessper
The Machmen
|
Posted - 2008.03.26 12:05:00 -
[8]
Great idea.
I'd love a copy of the app and the source if available please!
- - - - - -
EveHQ Character App | Item DB |
|
CCP Lingorm
C C P
|
Posted - 2008.03.26 12:07:00 -
[9]
As long as you do not modify the cache files then you are free to read them and write tools for them. Of course we do not support these tools (*grin*) and if we change the cache file structure of methods they may break, not that I see this happening but it is possible.
And to make perfectly clear. If you modify them then your client may break.
CCP Lingorm
CCP Quality Assurance
QA Engineering Team Leader
|
|
postal dude
Thundercats
RAZOR Alliance
|
Posted - 2008.03.26 13:05:00 -
[10]
Okay then, here it is. I release this under the GNU GPLv3.
Please note, this is only for reading the standings out of the cache, if you want to parse other cachefiles, this program is no use for you.
Usage: You have to point the app to your cache folder. It is somewhere in you application-data-folder (OS-specific). The folder you must browse to looks like this: "[...]\cache\MachoNet\87.237.38.200\189\CachedMethodCalls".
Then just press "Read Standings". Please note it reads the standings from the first standings-cache file it finds, so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)
Have fun with it ;)
eve_cache_standings-gpl.zip
---
|
|
|
Zathi Shaitan
Illiteracy Combatants
|
Posted - 2008.03.26 15:46:00 -
[11]
Someone please share a binary version...
----
" Several unconventional alliances where made at that point " - Hey CCP, "where" != "were".. you too, Brutus? http://loseloose.com/ |
Shinhan
Phoenix Knights
Dark Nebula Galactic Empire
|
Posted - 2008.03.26 15:52:00 -
[12]
Originally by: postal dude
Okay then, here it is. I release this under the GNU GPLv3.
Please note, this is only for reading the standings out of the cache, if you want to parse other cachefiles, this program is no use for you.
Usage: You have to point the app to your cache folder. It is somewhere in you application-data-folder (OS-specific). The folder you must browse to looks like this: "[...]\cache\MachoNet\87.237.38.200\189\CachedMethodCalls".
Then just press "Read Standings". Please note it reads the standings from the first standings-cache file it finds, so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)
Have fun with it ;)
eve_cache_standings-gpl.zip
Where is the binary? Not everybody has Microsoft Visual Studio 9.
For everybody else, here's excerpt from some comments in the source code:
'0x15 0x02 0x04' is the header for the id=>standings lookup table
0x05 => Alliance; 0x06 => Corp
'0x15 0x03 0x04' is the header for the id=>name/type lookup table
How long did it took you to find the headers and do you have any tips on how to find the headers for other data?
I wanna see if the Market Price History is cached... :)
--
Selling apples, 1 signature each.
ѼѼѼѼѼѼѼ |
postal dude
Thundercats
RAZOR Alliance
|
Posted - 2008.03.26 16:15:00 -
[13]
Edited by: postal dude on 26/03/2008 16:15:06
Originally by: Shinhan
Where is the binary? Not everybody has Microsoft Visual Studio 9.
Here you go: EveCacheStandings.exe
(GNU GPLv3 applies here too.)
Originally by: Shinhan
How long did it took you to find the headers and do you have any tips on how to find the headers for other data?
I wanna see if the Market Price History is cached... :)
Well, it took me a few hours. I am working on a decoder for all the cachefiles at the moment. I cant tell you if the market price history is cached yet, but you can try and look at the cache yourself, a few words are plaintext in them
---
|
Cori4n
principle of motion
Interstellar Alcohol Conglomerate
|
Posted - 2008.03.27 01:25:00 -
[14]
Originally by: CCP Lingorm
As long as you do not modify the file in question it is technically not against the EULA or TOS. It is only a cached version of a database method call.
How about the logserver? :D
|
MrRx7
Cutting Edge Incorporated
RAZOR Alliance
|
Posted - 2008.03.27 06:14:00 -
[15]
as alway I love you work.
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2008.03.28 00:39:00 -
[16]
Originally by: postal dude
As I understand there is no standings export via the API or the eve-online website.
I poked around in the eve-cache folders and stumbled over a file with the corp standings in it.
This approach is really interesting. I scanned through some of the cachefiles and tried to find patterns. It actually looks as if the cachefiles are some kind of object serialization code (likely Opcode/Stackbased) which means it should be possible to read them programmatically.
As far as I've seen this might make quite some of the live APIs mostly useless. In fact it seems you could get much more data from the cachefiles that from the API.
BTW: It seems most of the caches are extremely inefficiently implemented. No wonder Eve hammers the CPU...
|
Verite Rendition
F.R.E.E. Explorer
Atrum Tempestas Foedus
|
Posted - 2008.03.28 02:01:00 -
[17]
Originally by: Amida Ta
Originally by: postal dude
As I understand there is no standings export via the API or the eve-online website.
I poked around in the eve-cache folders and stumbled over a file with the corp standings in it.
This approach is really interesting. I scanned through some of the cachefiles and tried to find patterns. It actually looks as if the cachefiles are some kind of object serialization code (likely Opcode/Stackbased) which means it should be possible to read them programmatically.
As far as I've seen this might make quite some of the live APIs mostly useless. In fact it seems you could get much more data from the cachefiles that from the API.
BTW: It seems most of the caches are extremely inefficiently implemented. No wonder Eve hammers the CPU...
The API is a stable platform that can be accessed without a client however, the cache files are not. Plus the moment you use a macro to automate logging in to EVE to collect and extract data, you're violating the EULA. Still, this can definitely be useful in certain cases.
----
FREE Explorer
Lead Megalomanic
EVE Automated Influence Map |
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2008.03.28 08:19:00 -
[18]
Originally by: postal dude
Well, it took me a few hours. I am working on a decoder for all the cachefiles at the moment. I cant tell you if the market price history is cached yet, but you can try and look at the cache yourself, a few words are plaintext in them.
Edit: I quickchecked for that and after a few minutes I found the item-id, but not the price nor the quantity. But was just a quick view over the file.
Did you make any advances?
I seem to understand quite a lot that is within the files (at least as long as lists are used), however the header (the whole Class/Methodcall/etc. stuff is still pretty obscure.
Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...
|
Herio Mortis
54th Knights Templar
THORN Alliance
|
Posted - 2008.03.28 08:51:00 -
[19]
Originally by: Amida Ta
Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...
As far as I can tell that byte is an index into an array of strings. Seems to be used as a form of
compression for often used classnames etc.
|
Ssoraszh Tzarszh
The Royal Engineers
|
Posted - 2008.03.28 09:06:00 -
[20]
Originally by: postal dude
<great stuff>
so make sure there is no out-of-date cache file. (Is that even possible? I dont know if EVE deletes old files in there)
Have fun with it ;)
eve_cache_standings-gpl.zip
Yes, that is possible, i just got a standings list from a corp i left a few months ago :D
|
|
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2008.03.30 11:24:00 -
[21]
Anybody made any advances?
Would be especially interested in OpCodes:
&H2C (Objectstart?)
&H17 (Constructorcall?)
&H14 (List/Tuples?)
&H20
&H0D
|
Vessper
|
Posted - 2008.03.30 12:03:00 -
[22]
Originally by: Amida Ta
Anybody made any advances?
Would be especially interested in OpCodes:
&H2C (Objectstart?)
&H17 (Constructorcall?)
&H14 (List/Tuples?)
&H20
&H0D
Quoting from my list:
"0D" + len + string: string of length "len" followed by "string". len is byte, if len==255 then len followed of 4 bytes of longint len
"14" + len + data: Tuple of "len" elements followed by elements themselves.
"17" + a + b: Class "a" calling "b" method
Hope that's some help to you as I'm failing miserably at trying to decode these 
- - - - - -
 Please visit your user settings to re-enable images.
EveHQ Char App | Item DB |
Fubar
Eagle Eye Inc.
|
Posted - 2008.03.30 14:27:00 -
[23]
Originally by: Amida Ta
Did you alread find out what opcode &H11 is? I assumed it may be an empty ASCII-String, however it seems it is followed by another Byte which I could not find out...
Opcode &H11 seems to be a short cut for common names/tags.
This is the list I have compiled so far.
&H08 = bloodLineID
&H09 = capacity
&H0A = categoryID
&H15 = tickerID
&H18 = Description
&H22 = graphicsID
&H23 = groupID
&H2C = locationID
&H2D = locationName
&H3F = ownerID
&H41 = ownerName
&H42 = raceID
&H4A = typeID
|
Amida Ta
|
Posted - 2008.04.01 21:49:00 -
[24]
Here are some observations:
Byte 2E () = String (Unicode?)
Byte XX StringLen (if XX is FF another 4 bytes follow that specify the length)
XX Bytes String Chars
Byte 0E () = String empty (Unicode?)
Single Opcode
Byte 2F () = Datetime?
Byte XX Datalength (usually 08)
XX Bytes Date data
Byte 10 () = String (ASCII? ObjectName?)
Byte XX StringLen
XX Bytes String Chars
Byte 0F () = Char/String (ASCII? that is 1 char long)
Byte 1: Character
Byte 11 () = ASCII String? Link to static String Table?
1 Byte Index Data ?
Byte 01 () = Null/Empty?????
Single Opcode
Byte 02 () = String (ObjectName?)
Byte XX StringLen
XX Bytes String Chars
Byte 04 () = 4 Byte Int
4 Bytes Int Data
Byte 05 () = 2 Byte Int
2 Bytes Int Data
Byte 06 () = 1 Byte Int
1 Bytes Int Data
Byte 08 () = Int Value Fixed: 0 ??? (false)
Single Opcode
Byte 09 () = Int Value Fixed: 1 (true)
Single Opcode
Byte 28 () = ?
Single Opcode
Byte 0A () = 8 Byte Double?
8 Bytes Double Data
Byte 15 () = Datensatz? (Row?)
Byte XX Anzahl der folgenden DatensStze
Byte 16 () = Dataset
Byte XX Number of Dataset items following
Byte 2C () = Object-Start? Commit Data?
Single Opcode
11 43 steht einer Tabelle vor
11 2B folgt nach Tabelle
11 24 folgt nach Tabelle
11 3F = "name" ?
1B 01 ?
17 11 51 16 03 Tabellendefinition?
17
17 11 hSufig
17 11 36 14 07 2C (2F 08 = Header Date) kommt zu Beginn fast nberall vor
7E 00 00 00 00 Header?
Bytes 1-4: Version?
Byte 14 () = Methodcall ???? List?
Byte XX Number of Parameters/Definition ??
1. Parameter: Class
2. Parameter: Method
3+ Parameter: Arguments
Byte 17 = New Object/Constructor call
1. Param: Objectname
2. Param Methodcall (Byte 14)
16 01 62 2C ?
Header 1 : 7E 00 00 00 00
Objectstart: 2C
Methodstruct+Parameter/Objectname
Payloadheader:
17 11 36 14 07
Datetime (Updatetime?)
INT
Empty
INT
INT/Bool (Fixed: True)
20 2B FF alternative 0D FF 13 27
|
Illusori
Salvage and Mining Consortium
Kinetic Maelstrom Alliance
|
Posted - 2008.04.02 21:36:00 -
[25]
NIce work there Amida, seems to support what I've found,
using a file as follows:
00000000: 7e00 0000 002c 1403 100b 4d65 7468 6f64 ~....,....Method
00000010: 2043 616c 6c2e 0570 726f 7879 1404 1188 Call..proxy....
00000020: 1012 4765 744e 6577 5072 6963 6548 6973 ..GetNewPriceHis
00000030: 746f 7279 04c0 9698 0005 0667 1711 3614 tory.......g..6.
00000040: 072c 2f08 82e8 7152 258d c801 05d2 2201 .,/...qR%.....".
00000050: 04fd 460f 0009 2b89 7e00 0000 0023 2c25 ..F...+.~....#,%
00000060: 020e 6462 7574 696c 2e43 526f 7773 6574 ..dbutil.CRowset
00000070: 1601 222c 0214 626c 7565 2e44 4252 6f77 ..",..blue.DBRow
00000080: 4465 7363 7269 7074 6f72 2514 062c 100b Descriptor%..,..
00000090: 6869 7374 6f72 7944 6174 6506 402c 1008 historyDate.@,..
000000a0: 6c6f 7750 7269 6365 0606 2c10 0968 6967 lowPrice..,..hig
000000b0: 6850 7269 6365 0606 2c10 0861 7667 5072 hPrice..,..avgPr
000000c0: 6963 6506 062c 1006 766f 6c75 6d65 0614 ice..,..volume..
000000d0: 2c10 066f 7264 6572 7306 032d 2d11 242d ,..orders..--.$-
000000e0: 2d08 1403 100b 4d65 7468 6f64 2043 616c -.....Method Cal
000000f0: 6c2e 0570 726f 7879 1404 1188 1012 4765 l..proxy......Ge
00000100: 744e 6577 5072 6963 6548 6973 746f 7279 tNewPriceHistory
00000110: 04c0 9698 0005 0667 .......g
I've worked out the following meaning:
---- Header ----
?? # (7e 0000 0000) header?
key/value # (2c)
---------- block repeated at end ----------
[ # 3 (14 03)
'Method Call' # (10 0b ...)
'proxy' # (2e 05 ...)
[ # 4 (14 04)
?? # (11 88)
'GetNewPriceHistory' # (10 12 ...)
regionid:'Essence'
# (04 c0969800) # 009896c0 is essence regionid (10000064)
typeid:'capacitor control circuit II'
# (05 0667) # 6706 is typeid for t2 ccc (26374)
]
]
------------------ end block --------------
?? # (17 11 36)
[ # 7 (14 07)
key/value # (2c)
?? # (2f 08 82e8 7152 258d c801)
?? # (05 d222)
?? # (01)
?? # (04 fd46 0f00)
?? # (09)
?? # (2b89 7e00 0000 0023 2c25)
'dbutil.CRowset' # (02 0e ...)
?? # (16 01 22)
key/value # (2c)
'blue.DBRowDescriptor%' # (02 14 ...)
[ # 6 (14 06)
key/value # (2c)
'historyDate' # (10 0b ...)
40 # (06 40)
key/value # (2c)
'lowPrice' # (10 08 ...)
6 # (06 06)
key/value # (2c)
'highPrice' # (10 09 ...)
6 # (06 06)
key/value # (2c)
'avgPrice' # (10 08 ...)
6 # (06 06)
key/value # (2c)
'volume' # (10 06 ...)
14 # (06 14)
key/value # (2c)
'orders' # (10 06 ...)
3 # (06 03)
]
?? # (2d2d 1124 2d2d 08)
---------- block repeated at start ----------
[ # 3 (14 03)
'Method Call' # (10 0b ...)
'proxy' # (2e 05 ...)
[ # 4 (14 04)
?? # (11 88)
'GetNewPriceHistory' # (10 12 ...)
regionid:'Essence'
# (04 c0969800) # 009896c0 is essence regionid (10000064)
typeid:'capacitor control circuit II'
# (05 0667) # 6706 is typeid for t2 ccc (26374)
]
]
------------------ end block --------------
From that I'd basically come to the same conclusions as Amida, except that 0x2c to me appears to always indicate a key/value pair, or at least a two-element array.
I'm strapped for time at the moment, but hopefully this is useful for others to take things forwards.
'Sori
|
Pasion Helada
SPANI
T e r c i o s
|
Posted - 2008.05.13 16:01:00 -
[26]
Edited by: Pasion Helada on 13/05/2008 16:01:41
all lengths are 1 byte and if byte == 255 another 4 follow.
16 is a dictionary
followed by len
( value / key ) * len times
2C is a tuple of 2 items
25 is a tuple of 1 item
24 is an empty tuple
26 is an empty sequence
1C is always followed by 4 bytes, but unknown meaning at the moment.
|
|
|
| |
|
| Pages: [1] :: one page |
| First page | Previous page | Next page | Last page |