Pages: 1 2 [3] 4 5 6 7 8 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
YunFu Yan
Yan Enterprises
|
Posted - 2009.02.24 23:33:00 -
[61]
Eve metrics does not use simple maths to generate the statistics.
How I know? Well, I kinda contributed to the code: Outliers are identified and eliminated before doing any relevant calculations.
This project has a lot of potential. Especially so once we users get the means to work with the underlying data warehouse! ------------------------------------------------- Yan Enterprises - We mean business. |
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.02.24 23:41:00 -
[62]
Edited by: Ix Forres on 24/02/2009 23:42:47
Originally by: Hexxx Tell us what your needs are and we'll make it happen.
edit: to be more clear, there are people here that can make sure you get the kind of heavy lifting you need out of a proper host
I run the server myself; it's a dedicated box I rent. It's more than capable of running EM, but there was some hugely unoptimised code in there, and absolutely no caching whatsoever anywhere. tl;dr, I just tightened it all up and now we're golden. All the hard work is done by background workers (more or less); if it needs to expand, then I already have offers from several people for CPU time on their boxes. The workers use a central queue, so it's a simple matter to add capacity- I even have it set up so I can use Amazon EC2 to add capacity dynamically, though this isn't enabled yet.
Originally by: Akita T Edited by: Akita T on 24/02/2009 19:33:57
__
Also, I noticed something weird with the uploader so I'll have to check with you...
I have the "delete after upload" selected (you know, to ease the load in case I want to do a "force upload"), and did quite a bunch of in-game browsing. Obviously, there was quite a bit of timeouts and so on and so forth, and I also accidentally closed the uploader (got used with evemon and the fact it only closes from tray), tried to do a "force upload" for the rest of thingies not up yet due to the server crash... but now the uploader says it finds nothing (yet there should be plenty not yet uploaded)... plus, in-game, I still get the cached data on the history graph (no delay on load as opposed to a second or so delay on first uncached load). Will the uploader eventually recognize the cached data or not ? Of course, we could simply log off, wipe the EVE cache, start everything from scratch as a "worst case" scenario, but... meh.
So the question is, how does the uploader handle all of this, and what should OUR "best practice" be in such cases ? ___
P.S. Have you thought collaborating with Chribba on this one ? I think he'd love this as an addition to his arsenal of useful stuff, and you could surely use the CPU firepower he might be able to provide... sure, there would be a lot of things to discuss and probably a lot of compromises to be made, but I think it at least deserves some degree of consideration.
If the uploader can't find anything make sure all your EVE installs are set properly. If you're using a different cache path (some launchers can modify this), see if it's being detected in the log window. If it's not, try it without the launcher. The uploader guesses your cache paths from your EVE install locations, which should work in the vast majority of cases.
As to any collaboration with Chribba: This has always been an independent project and for now it'll probably stay that way. That said, I hope to open EVE Metrics up as it progresses with more APIs, so if Chribba wants to get into marketeering and this sort of analysis he's more than welcome to use the APIs. I've not approached him about any sort of partnership and vice versa; there's no real need or demand.
On the statistics note: No, they're not simple averages. It's a combination of quartile range filtering to remove outliers, along with filtering based on standard deviation and variance to further remove crap orders. The filters are run once for price and once for quantity. The results are summed and those orders are then used to generate an average. More or less, anyway; it varies across the site. ERAM/ERAP indexes use the number direct from CCP's own output, EMCP is what EVE Metrics can deduce given the orders it has to work with. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Akita T
Caldari Navy Volunteer Task Force
|
Posted - 2009.02.25 00:05:00 -
[63]
Originally by: Ix Forres If the uploader can't find anything make sure all your EVE installs are set properly. If you're using a different cache path (some launchers can modify this), see if it's being detected in the log window. If it's not, try it without the launcher. The uploader guesses your cache paths from your EVE install locations, which should work in the vast majority of cases.
On the previous "weird stuff" report : single install, manual client location, cache folders displayed by the uploader are always the proper ones (it even detects new cache hits), no launchers used. The issue was the combination between server unavailable and uploader close/reopen while EVE was still running normally, not taking "old" caches (that is same EVE session but different uploader session) into acount, with "delete cache file after upload" being an unknown factor (EVE *seemed* to still have the very same data already cached, but uploader doesn't detect the cache file as "upload-worthy").
_ Create a character || Fit a ship || Get some ISK |
Namco
Balls of Steel
|
Posted - 2009.02.25 15:30:00 -
[64]
This is all I see in the log window:
Unable to Cast Type: The specified module could not be found. (Exception from HRESULT: 0x8007007E) at .Initialize() at TrayUploader.Form1.Register(IPlugin ipi) at .set_Host(IPluginHost ) at TrayUploader.Form1.Start(String args) Unable to load assembly: ManagedZLib The computer is running Vista64.
|
J'i Ta
|
Posted - 2009.02.25 18:29:00 -
[65]
Originally by: Namco This is all I see in the log window:
Unable to Cast Type: The specified module could not be found. (Exception from HRESULT: 0x8007007E) at .Initialize() at TrayUploader.Form1.Register(IPlugin ipi) at .set_Host(IPluginHost ) at TrayUploader.Form1.Start(String args) Unable to load assembly: ManagedZLib The computer is running Vista64.
That's the packed/obfuscated cache scraping code failing to load.
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.02.25 19:22:00 -
[66]
Originally by: J'i Ta
Originally by: Namco This is all I see in the log window:
Unable to Cast Type: The specified module could not be found. (Exception from HRESULT: 0x8007007E) at .Initialize() at TrayUploader.Form1.Register(IPlugin ipi) at .set_Host(IPluginHost ) at TrayUploader.Form1.Start(String args) Unable to load assembly: ManagedZLib The computer is running Vista64.
That's the packed/obfuscated cache scraping code failing to load.
J'i Ta- is your PC running a 64-bit OS too? That might be what's causing it- if it is, we'll get a 64-bit build up and available ASAP. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
BlindBleu
Gallente Freedom-Technologies
|
Posted - 2009.02.26 00:25:00 -
[67]
I am unable to get the web page to load, using firefox or IE.m Had this happen last night, worked this morning, now it does not. Every Plan lasts until the first shot is fired. "I have a Plan, a good one." Generally means it is not. If Yellow it get a Fellow Killed |
Niedar
MASS
|
Posted - 2009.02.26 03:29:00 -
[68]
Edited by: Niedar on 26/02/2009 03:30:12
Originally by: Ix Forres
J'i Ta- is your PC running a 64-bit OS too? That might be what's causing it- if it is, we'll get a 64-bit build up and available ASAP.
I get the same error and also run Windows Vista 64.
|
Akita T
Caldari Navy Volunteer Task Force
|
Posted - 2009.02.26 04:59:00 -
[69]
I noticed something else that's somewhat strange.
When viewing ORDER data, usually the uploader reports that TWO logfiles have been uploaded, and viewing market history data on the same item does not increase the count of logfiles uploaded. Also, if viewing history first, it's just one logfile uploaded, then when viewing orders it's another one. So, this leads me to believe that the history data is loaded into cache even when just viewing orders, so technically viewing the history data would not be necessary. HOWEVER, on some items I've uploaded data from up to three different regions (by doing order viewing first, then either history afterwards or no history at all), the ERAM/ERAP was "unavailable", so this leads me to believe the history data might actually not be loaded (or uploaded) correctly.
So, which one is it ?
_ Create a character || Fit a ship || Get some ISK |
Verite Rendition
Caldari F.R.E.E. Explorer Wildly Inappropriate.
|
Posted - 2009.02.26 05:07:00 -
[70]
Originally by: Niedar Edited by: Niedar on 26/02/2009 03:30:12
Originally by: Ix Forres
J'i Ta- is your PC running a 64-bit OS too? That might be what's causing it- if it is, we'll get a 64-bit build up and available ASAP.
I get the same error and also run Windows Vista 64.
FWIW I have no issues on Vista 64. ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.02.26 08:14:00 -
[71]
Hi, nice project ;)
However I have some issues/questions/suggestions:
1) I tried using the developer APIs, and they seem to be extremely slow (waited >3 min for a quote for a single product type)
2) I find it somewhat funny that we both developed a cache decryptor, I got in touch with quite some people to do so, but I cannot remember you ;)
3) Why do you obfuscate your code? It looks as you may have something to hide.
4) The ZLib you are using is a mixed mode assembly which is bad because it prevents to run on linux and might create problems on 64bit windows.
5) I might add eve-metrics as MarketAPI to EveAI.Live.
6) Is it possible to package your upload-utility without a setup .exe file. I (and probably others) have a huge distrust for such kind of executables and it is likely completely unneccessary. E.g. use a simple .ZIP instead.
|
Caladain Barton
Vengeance Imperium
|
Posted - 2009.02.26 08:30:00 -
[72]
Originally by: Amida Ta Edited by: Amida Ta on 26/02/2009 08:20:16 Hi, nice project ;)
However I have some issues/questions/suggestions:
1) I tried using the developer APIs, and they seem to be extremely slow (waited >3 min for a quote for a single product type)
2) I find it somewhat funny that we both developed a cache decryptor (in C#?), I got in touch with quite some people to do so, but I cannot remember you ;)
3) Why do you obfuscate your code? It looks as you may have something to hide.
4) The ZLib you are using is a mixed mode assembly which is bad because it prevents to run on linux and might create problems on 64bit windows.
5) I might add eve-metrics as MarketAPI to EveAI.Live.
6) Is it possible to package your upload-utility without a setup .exe file. I (and probably others) have a huge distrust for such kind of executables and it is likely completely unneccessary. E.g. use a simple .ZIP instead.
Any chance either of you guys/gals would be willing to write up a wiki page detailing how you solved the issue/how the cache is actually formatted? I haven't spent a lot of time looking or poking at the issue, but it does appear to be a good case of decryption and from a purely programmatic or mathematical point of view, i would find such an article quite a fun and interesting read.
Or maybe i'm just naive and silly..
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.02.26 08:34:00 -
[73]
Originally by: Caladain Barton
Any chance either of you guys/gals would be willing to write up a wiki page detailing how you solved the issue/how the cache is actually formatted? I haven't spent a lot of time looking or poking at the issue, but it does appear to be a good case of decryption and from a purely programmatic or mathematical point of view, i would find such an article quite a fun and interesting read.
Or maybe i'm just naive and silly..
CCP already said that reading these files would be ok, but that they would not like to see any knowledge spread around how do decode these (likely because if somebody can decode them the way to modify them is not too long and that might open a whole lot of problems for CCP)
|
Caladain Barton
Vengeance Imperium
|
Posted - 2009.02.26 08:41:00 -
[74]
Originally by: Amida Ta
Originally by: Caladain Barton
Any chance either of you guys/gals would be willing to write up a wiki page detailing how you solved the issue/how the cache is actually formatted? I haven't spent a lot of time looking or poking at the issue, but it does appear to be a good case of decryption and from a purely programmatic or mathematical point of view, i would find such an article quite a fun and interesting read.
Or maybe i'm just naive and silly..
CCP already said that reading these files would be ok, but that they would not like to see any knowledge spread around how do decode these (likely because if somebody can decode them the way to modify them is not too long and that might open a whole lot of problems for CCP)
Ahh, quite sad then. I originally was looking at it for exactly the purposes you guys put it to use for, automatic market scraping for my own personal calculations, but gave up on it (admittedly not trying for very long) as it became obvious that it would require a good chunk of my time (which, while in university, i simply just can't justify). I imagine the story/algorithm/decryption must be a very fun story :-)
|
destinationunreachable
|
Posted - 2009.02.26 12:35:00 -
[75]
I like the application and are happy to hear, that CCP has given their blessing. Even though the web interface still needs some cleanup, I predict it will be more successful than eve-central as you upload on-the-fly, which out pressing the bloody export button. I would like to do some stats calculation - would it be possible to get a full (nightly?) database dump ?
|
Agrapena
|
Posted - 2009.02.26 12:45:00 -
[76]
^5s to an amazing piece of software. the ammount of time saved by not hitting that export button is phenominal, specially if you`re in Jita!!
Previous posters mentioned the web interface needing tidied up, which is agreeable, but at the same time, it still serves its purpose and does it well.
Definately a welcome change to Eve Central.
|
FireFoxx80
Caldari E X O D U S Imperial Republic Of the North
|
Posted - 2009.02.26 14:34:00 -
[77]
Ooo, this does look nifty.
However, I thought that CCP tried to stamp this behaviour out? The whole BACON scandal where people were using sikrit logserver data to alert you to hostiles in the system caused CCP to modify the way that logserver handled data; yet the cache is just the same way of accessing the same data.
So in a way, CCP haven't actually prevented people from accessing sikrit data, they've just made it harder?
What I do the rest of the time |
Akita T
Caldari Navy Volunteer Task Force
|
Posted - 2009.02.26 15:13:00 -
[78]
It's different data though. BACON alerted you of something time-critical when NOT on the computer (or not paying attention), giving you a smidgen of an unfair advantage towards the others not using it. This here serves just to make existing functionality less time-consuming, and does nothing you can't already do... it won't automatically browse the markets for deals, it only collates the data you manually select to view anyway.
_ Create a character || Fit a ship || Get some ISK |
Emily Spankratchet
Minmatar Pragmatics
|
Posted - 2009.02.26 16:29:00 -
[79]
This isn't open source, and I'm paranoid about running anything that might be a keylogger on my gaming machine. Has anybody looked at this to make sure it does what it says it does and nothing else? I can see any mention of the issue in this thread.
I'm somewhat convinced by Hexxx vouching for it, given that I at least trust ebank with my ISK. More reassurance would be nice, though.
|
Business Ethics
|
Posted - 2009.02.26 16:51:00 -
[80]
Did all common sense go out the window with this thread?
If you're an active and profitable trader than what are you doing running an executable that actively sc****s your cache and uploads the data to a 3rd party site? Talk about giving away the baby with the bathwater along with the whole bathroom.
Notice any extra competition in your markets since you started running this? It's trivial for these guys to mine the uploads for profitable markets. You think they're doing this out of a sense of community?
Until the source code for this uploader is made available and vetted anyone running this is playing with fire. Not to mention that CCP hasn't come in and given it a EULA-compliant thumbs up yet.
Jeez I hate to be negative nancy here but why would anyone with a secret recipe for profit just upload it to the internet?
|
|
Akita T
Caldari Navy Volunteer Task Force
|
Posted - 2009.02.26 17:04:00 -
[81]
Originally by: Business Ethics If you're an active and profitable trader than what are you doing running an executable that actively sc****s your cache and uploads the data to a 3rd party site?
Only run it if/when you have stuff you need uploaded and/or wish to contribute. Personally, I simply browsed several "interesting to watch for effects but not necessarily good to trade in directly" WHOLE CATEGORIES of items in several different regions. I manually kill my "machonet" folder regularly anyway, it doesn't hurt me to spend one hour or so once a month doing some bulk uploads for the most interesting item categories in 3 neghbouring regions.
As for probability the uploader is a trojan/keylogger/whatever... well, I doubt this particular set of people would be willing to risk a banning on account of doing something like that... but yeah, there's always that risk, even if it's minimal. Sure, I'd love to see it go open-source, but then again... how many of you ACTUALLY looked at the FULL code of an open source project ? How many people you know that looked into the complete source code of the invention calculator, or the source code for EVE-Mon that aren't involved in its development in the first place ? Or, what about EFT, fercrissakes ? Oh, what about the many macro-enabled, password-to-fully-view-protected XLS pseudo-calculators around ?
Sometimes, a little bit of risk is necessary for something nice.
_ Create a character || Fit a ship || Get some ISK |
Business Ethics
|
Posted - 2009.02.26 17:19:00 -
[82]
Well what we need is some sponsored alts that sit in each region and upload the whole market once a day. This would probably be worth isk to some people. How many items (total) are listed on the market anyway?
It's just so tough to piece together a viable picture of the market like this. It's like looking at the grand canyon through a toilet-paper tube.
Honestly I'd rather CCP give us something like a Technical Analysis skill that opened up different levels of API data for the local region's market (for that char). Level 1 might give you access to basic (daily) tick data at your local station, all the way up to full regional market dumps, tick data, and all the other crap, at level 5.
Just fantasizing here.
|
Nick Turing
|
Posted - 2009.02.26 17:41:00 -
[83]
Originally by: Emily Spankratchet This isn't open source, and I'm paranoid about running anything that might be a keylogger on my gaming machine. Has anybody looked at this to make sure it does what it says it does and nothing else? I can see any mention of the issue in this thread.
I'm somewhat convinced by Hexxx vouching for it, given that I at least trust ebank with my ISK. More reassurance would be nice, though.
I've looked through the assemblies in Reflector fairly thoroughly, and I can state with a high degree of certainty that this does not contain a keylogger or anything else other than what's on the box.
Obviously, you've got no particular reason to trust _me_ either, but anyone with Reflector and some knowledge of .NET can verify this, despite the obfuscation applied to one of the assemblies.
|
Nick Turing
|
Posted - 2009.02.26 17:47:00 -
[84]
Originally by: Business Ethics Did all common sense go out the window with this thread?
If you're an active and profitable trader than what are you doing running an executable that actively sc****s your cache and uploads the data to a 3rd party site? Talk about giving away the baby with the bathwater along with the whole bathroom.
Notice any extra competition in your markets since you started running this? It's trivial for these guys to mine the uploads for profitable markets. You think they're doing this out of a sense of community?
Until the source code for this uploader is made available and vetted anyone running this is playing with fire.
If you're concerned about 'them' using market data you upload to your detriment, why would seeing the source make any difference? Your concern seems to be with what it's claimed to do, not with any issues of "maybe it does something else". And given the public nature of the uploaded data, why are you concerned specifically about the authors of the tool? Anyone can access the data you upload - that's the point.
Quote: Not to mention that CCP hasn't come in and given it a EULA-compliant thumbs up yet.
Quoth CCP Lingorm: Quote: As long as you do not modify the cache files then you are free to read them and write tools for them. Of course we do not support these tools (*grin*) and if we change the cache file structure of methods they may break, not that I see this happening but it is possible.
And to make perfectly clear. If you modify them then your client may break.
|
Business Ethics
|
Posted - 2009.02.26 18:29:00 -
[85]
Originally by: Nick Turing
If you're concerned about 'them' using market data you upload to your detriment, why would seeing the source make any difference? Your concern seems to be with what it's claimed to do, not with any issues of "maybe it does something else". And given the public nature of the uploaded data, why are you concerned specifically about the authors of the tool? Anyone can access the data you upload - that's the point.
2 separate but equally valid concerns here. Is the executable itself safe (probably) and, do you want to share your data?
Quote: Quoth CCP Lingorm
ok cool then
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.02.26 21:48:00 -
[86]
Originally by: Business Ethics Did all common sense go out the window with this thread?
If you're an active and profitable trader than what are you doing running an executable that actively sc****s your cache and uploads the data to a 3rd party site? Talk about giving away the baby with the bathwater along with the whole bathroom.
Notice any extra competition in your markets since you started running this? It's trivial for these guys to mine the uploads for profitable markets. You think they're doing this out of a sense of community?
Until the source code for this uploader is made available and vetted anyone running this is playing with fire. Not to mention that CCP hasn't come in and given it a EULA-compliant thumbs up yet.
Jeez I hate to be negative nancy here but why would anyone with a secret recipe for profit just upload it to the internet?
Speaking personally; yes, I do it out of a sense of community. I write tools that could be useful and share them. I get a kick out of helping other people do cool stuff. Of course, EVE Metrics may expand in the future to bring paid-for tools (We're thinking about providing corporate tools for industry under the site, which would be a paid-for service). We're also thinking about the suggested pay-for-uploads strategy to reward uploaders, but we'd need to be turning a profit to do that!
Originally by: Emily Spankratchet This isn't open source, and I'm paranoid about running anything that might be a keylogger on my gaming machine. Has anybody looked at this to make sure it does what it says it does and nothing else? I can see any mention of the issue in this thread.
I'm somewhat convinced by Hexxx vouching for it, given that I at least trust ebank with my ISK. More reassurance would be nice, though.
We cannot publish the source to our uploader. I'd love to, but it's been made clear that we cannot. I've lost count of the number of people who have sc****d through with disassemblers, reflectors, and sc****d all the network traffic; it's clean, legit, has no nasties in it, and so on. None of the traffic is encrypted; it's all in the clear, so you can go through yourself and look at what we get. It carries a cryptographic hash to prevent tampering, but that's all. I will be making a .zip of the application available for those who don't want to trust a simple installer.
Other comments... yep, the web interface could use some tidying up. When I get a spare moment. BACON was a logserver exploit. EMU is a cache reader, which is a different thing altogether. Cache reading has been given the thumbs-up from CCP in regard to the EULA (See the quote from Lingorm), and we've run this by CCP (at a source level) to make sure they're fine with it.
In regard to us mining data... well, that's kinda what the site is for ;) your market orders are entirely public in any case; if this just makes it easier for people to see what you're doing then you probably want to rethink your strategy. Any strategy which depends on people being ignorant is probably not gonna fare too well.
Originally by: Amida Ta Edited by: Amida Ta on 26/02/2009 08:30:11 Hi, nice project ;)
However I have some issues/questions/suggestions:
1) The server's having issues; we're working on it. 2) Yes, C#. It's independently developed. 3) We obfuscate our code because we are aware that CCP does not everyone and their dog to have a cache reader. Not our decision. 4) Yes, sorry. We're looking at alternatives for Linux, and will certainly be making a logfile uploader (no cache reader) available for Linux and OSX in the coming months, if not a full-blown cache reader. 5) Cool! More usage is always good. 6) Will be doing this for the next version. 7) Suggest away as to what you'd like to see: there's a 'feedback' button on the site where you can suggest new ideas and vote on other people's suggestions. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Nick Turing
|
Posted - 2009.02.26 22:10:00 -
[87]
Originally by: Ix Forres
3) We obfuscate our code because we are aware that CCP does not everyone and their dog to have a cache reader. Not our decision.
Given that, to my understanding, the concern is not so much about people having cache readers as people being able to code cache _writers_, have you considered exposing more of the API? I realise it only implements a limited set of opcodes, but in general, being able to parse a cache file and get back a data structure of some kind would be rather handy, and wouldn't - as far as I'm aware - make CCP unhappy.
|
Modescond
|
Posted - 2009.02.26 22:16:00 -
[88]
for those of you who have loader issues on Vista x64.
Goto the following location: C:\Program Files (x86)\EVE Metrics\Uploader
right click "TrayUploader.exe", goto Details and tell me what version you have.
Also, run the application, open up task manager (ctrl+shift+esc), select the Processes tab and let me know if there is a * next to "TrayUploader.exe".
Thanks,
|
Andrea Griffin
|
Posted - 2009.02.26 23:00:00 -
[89]
I like this tool much better than the Eve Central Market Uploader - having to click the export button was annoying enough that I stopped after a while. EMU is great; start it up and forget about it. Thanks for your hard work. Looking forward to the mountains of data that will be showing up. <3
My only issue is that the website is pretty slow for me, and sometimes the uploader has timeout issues when trying to toss data back up to you guys. Hope this can be rectified soon.
|
Namco
Balls of Steel
|
Posted - 2009.02.26 23:16:00 -
[90]
Originally by: Modescond Edited by: Modescond on 26/02/2009 22:25:45 for those of you who have loader issues on Vista x64.
Goto the following location: C:\Program Files (x86)\EVE Metrics\Uploader
right click "TrayUploader.exe", goto Details and tell me what version, file-size and last modified date you have.
Also, run the application, open up task manager (ctrl+shift+esc), select the Processes tab and let me know if there is a * next to "TrayUploader.exe".
Thanks,
Version: 1.1.0.0 Size: 159KB Date Modified: 11/10/2008 5:25AM
Shows *32 next to the process in Task Manager.
|
|
|
|
|
Pages: 1 2 [3] 4 5 6 7 8 :: one page |
First page | Previous page | Next page | Last page |