Pages: 1 2 3 [4] 5 6 7 8 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Business Ethics
|
Posted - 2009.02.26 23:17:00 -
[91]
Originally by: Ix Forres but we'd need to be turning a profit to do that!
You have prime IPO material here. Get a math/statistics guy on staff analysing the database and hire a couple auditable traders and you will be all be trillionaires within a year, not kidding. I bet you could keep 2 dozen trade alts busy full time with the output from such a system.
Quote:
We cannot publish the source to our uploader.
Quote:
In regard to us mining data... well, that's kinda what the site is for ;)
You are in a unique position of having the database and also having the web server logs. If someone uploads juicy cache dumps you guys will notice it first!
Quote:
your market orders are entirely public in any case;
How is this Surely you'll have no problem telling me what what orders I have open if they're entirely public.
|
Caladain Barton
Vengeance Imperium
|
Posted - 2009.02.26 23:20:00 -
[92]
Originally by: Nick Turing
Originally by: Ix Forres
3) We obfuscate our code because we are aware that CCP does not everyone and their dog to have a cache reader. Not our decision.
Given that, to my understanding, the concern is not so much about people having cache readers as people being able to code cache _writers_, have you considered exposing more of the API? I realise it only implements a limited set of opcodes, but in general, being able to parse a cache file and get back a data structure of some kind would be rather handy, and wouldn't - as far as I'm aware - make CCP unhappy.
Maybe making the actual "decoder" a blackbox binary file, with fully documented interface for the rest of us to use? I would love to adapt this to my own programs, but i can understand if you don't want to/can't share how you did what you did in specific terms (documents/formulas/etc).
|
Fitz VonHeise
Foundation
|
Posted - 2009.02.26 23:35:00 -
[93]
So basically to populate your database, (after loading the program) all we have to do is have the market window open while whe fly from one region to the next.
|
Caladain Barton
Vengeance Imperium
|
Posted - 2009.02.26 23:42:00 -
[94]
Originally by: Fitz VonHeise So basically to populate your database, (after loading the program) all we have to do is have the market window open while whe fly from one region to the next.
I think this is the part of the equation i would like to remove..call me anti-social, but i like to keep to myself mostly, and i like to keep my data to myself. Perhaps some sort of licensing scheme to "purchase" the blackbox decoder?
I'm probably going off topic, but i view evemetrics as two items, one being the website and data being generated there (very cool by the way), and the second being this decoder, which i would love to see integrated into all the various tools i utilize at some point :-)
|
Verite Rendition
Caldari F.R.E.E. Explorer Wildly Inappropriate.
|
Posted - 2009.02.27 06:19:00 -
[95]
Originally by: Ix Forres
3) We obfuscate our code because we are aware that CCP does not everyone and their dog to have a cache reader. Not our decision.
If at some point it interests you, I'd like to talk to you (or whoever wrote the cache-scraping stuff) about putting together an application for me as an ISK-paid job. I've always wanted to do a version of my influence map for the Chinese EVE server (Serenity) but they do not have an API server like we do. I would need to sc**** the cache files for the sovereignty data and the alliance list, and while Paladin Vent figured this out, he did not leave behind the code to do this and I do not have the kind of knowledge needed to implement this myself.
If it's something you're interested in, drop me an EVEmail and we can talk. ---- FREE Explorer Lead Megalomanic EVE Null-Sec Player Influence Map http://dl1.eve-files.com/media/corp/Veritefw/FWinf |
Nick Turing
|
Posted - 2009.02.27 08:52:00 -
[96]
Edited by: Nick Turing on 27/02/2009 08:55:34
Originally by: Caladain Barton
Originally by: Nick Turing
Originally by: Ix Forres
3) We obfuscate our code because we are aware that CCP does not everyone and their dog to have a cache reader. Not our decision.
Given that, to my understanding, the concern is not so much about people having cache readers as people being able to code cache _writers_, have you considered exposing more of the API? I realise it only implements a limited set of opcodes, but in general, being able to parse a cache file and get back a data structure of some kind would be rather handy, and wouldn't - as far as I'm aware - make CCP unhappy.
Maybe making the actual "decoder" a blackbox binary file, with fully documented interface for the rest of us to use? I would love to adapt this to my own programs, but i can understand if you don't want to/can't share how you did what you did in specific terms (documents/formulas/etc).
It already is (minus the documentation), in the form of CRUDE.dll. The interface could use a bit of work, and it won't be able to decode the whole cache, but it's there.
Quote: So basically to populate your database, (after loading the program) all we have to do is have the market window open while whe fly from one region to the next.
You'll still need to view the individual items - it doesn't upload the entire market, just the bits of it you view.
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.02.28 23:26:00 -
[97]
Well, I'm happy to announce we've just moved to a brand spanking new backend server architecture which should help a lot with the performance of the app and general stability of things. We're also finishing up our testing of a new server environment provided by EVSCO which will be providing additional application server resources should we need it.
After these changes and additional under-the-hood tweaking we hope to have the whole app tuned much better and performing reliably well.
As to people suggesting we IPO this or start doing private projects: Frankly, there's more to life than ISK. Seriously! For now, I'm quite happy to do this for ISK donations as people see fit. We'll be introducing paid-for features (ISK payment ofc, not RL ISK) at some point in the future for the more complex and powerful tools we have planned but that's pretty far off right now on the timeline.
At the person wanting to know how to use the uploader: Yep, you just need to fly around and browse the market. Every item you view for which there's a delay before you see data, we get the data for (as in, not precached). Data uploaded is regionally focused, ofc.
I'm compiling some statistics on how we've grown in the past week which I'll throw up on the site later on. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Namco
Balls of Steel
|
Posted - 2009.02.28 23:31:00 -
[98]
Any chance of getting the cache scanner to work for those people (like myself) it doesn't work for?
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.02.28 23:45:00 -
[99]
Originally by: Namco Any chance of getting the cache scanner to work for those people (like myself) it doesn't work for?
I'm literally watching Salvis write the fix right now. So, yep. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Vaerah Vahrokha
Minmatar Dark-Rising
|
Posted - 2009.03.01 00:40:00 -
[100]
Quote:
Well, I'm happy to announce we've just moved to a brand spanking new backend server architecture which should help a lot with the performance of the app and general stability of things.
I just tried it, it's slow like crazy past the first screens.
|
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.03.01 01:07:00 -
[101]
Originally by: Vaerah Vahrokha
Quote:
Well, I'm happy to announce we've just moved to a brand spanking new backend server architecture which should help a lot with the performance of the app and general stability of things.
I just tried it, it's slow like crazy past the first screens.
Yeah; the new backend architecture just fell over. Give us a few hours to work out the kinks; I can't afford a staging server, unfortunately -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Salvis Tallan
Gallente The Shadow Order New Eden Research
|
Posted - 2009.03.01 01:14:00 -
[102]
A few things on the uploader. The code I wrote is compiled to run as 32-bit (so 64-bit will emulate that and work fine). The problem is the zlib library that is used to compress/decompress some stuff is not compiled that way and dies on 64-bit systems. I am currently working to replace it with one that plays nicer with others.
Second, The options for deleting files after upload or older than a certain date applies to exported market logs, as in the *.txt files that you get by clicking export on the market screen. It does not delete he cache files (because we do not in any way edit them), nor does 'force upload' work on said cache files.
Normally Ix and I are all for open-source, but in this case it will continue to be closed and obfuscated. If our assurance for the safty of the code still leaves you with doubts, simply delete the CRUDE dll and the app will no longer have the cache functionality. ------
|
Akita T
Caldari Navy Volunteer Task Force
|
Posted - 2009.03.01 06:12:00 -
[103]
By the way, feel free to include THIS here as part of your site. Actually, come to think of it, please, DO add most of the functionality there in your site... it should be awesome since you have a direct feed to buy/sell order prices, so all you need to do is take the rest of the raw data from that, enter the calculations (which are pretty simple) and voila, instant awesomeness^10.
_ Create a character || Fit a ship || Get some ISK |
Nick Turing
|
Posted - 2009.03.01 10:19:00 -
[104]
Originally by: Salvis Tallan A few things on the uploader. The code I wrote is compiled to run as 32-bit (so 64-bit will emulate that and work fine). The problem is the zlib library that is used to compress/decompress some stuff is not compiled that way and dies on 64-bit systems. I am currently working to replace it with one that plays nicer with others.
There's actually zlib support buried somewhere in the .NET sdk. The most obvious classes all expect the gzip header, but there's one buried in there somewhere that doesn't. Let me know if you're interested and can't find it.
Quote: Normally Ix and I are all for open-source, but in this case it will continue to be closed and obfuscated. If our assurance for the safty of the code still leaves you with doubts, simply delete the CRUDE dll and the app will no longer have the cache functionality.
You could always open-source the app, but not the plugin. Some clear terms on what you think about other people using CRUDE.dll for their own apps wouldn't hurt, either. :)
|
Khrillian
Minmatar Sebiestor tribe
|
Posted - 2009.03.01 10:55:00 -
[105]
Are we still limited to 20 queries per day? I know you can query multiple records at once with commas, but it's a tad easier for me to do one at a time.
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.03.01 12:51:00 -
[106]
Originally by: Khrillian Are we still limited to 20 queries per day? I know you can query multiple records at once with commas, but it's a tad easier for me to do one at a time.
Nope. This was actually only a bug with the documentation; I'd taken the limit out ages ago.
The site's currently being given a good going over- I'm in the process of refactoring one or two bits to make them easier to cache. As a result some parts are behaving erratically and we've had to disable indexes until we can sort out a better way to generate them, as they currently require too much memory to compute. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
ingenting
Cohors Alaria
|
Posted - 2009.03.01 13:20:00 -
[107]
trying to view full list of sell orders on tritanium = constant loading of the page _________________ - "Welcome to EVE, remember to insu *BAAOOM*... Told you, newb."
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.03.01 13:37:00 -
[108]
Originally by: ingenting trying to view full list of sell orders on tritanium = constant loading of the page
Odd- working fine here. I've just thrown a fix up to disable the click-order-for-details popup on the order pages if there's a lot of orders- see if that helps. The javascript might be slowing your browser down. -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Khrillian
Minmatar Sebiestor tribe
|
Posted - 2009.03.01 13:37:00 -
[109]
Originally by: Ix Forres
Originally by: Khrillian Are we still limited to 20 queries per day? I know you can query multiple records at once with commas, but it's a tad easier for me to do one at a time.
Nope. This was actually only a bug with the documentation; I'd taken the limit out ages ago.
The site's currently being given a good going over- I'm in the process of refactoring one or two bits to make them easier to cache. As a result some parts are behaving erratically and we've had to disable indexes until we can sort out a better way to generate them, as they currently require too much memory to compute.
Thanks.
I'm not too familiar with generating indicies. What do you want yours to be beyond a simple weighted average? Is most of the problem cleaning up the data before you average it and weighting multi-region indices?
|
Salvis Tallan
Gallente The Shadow Order New Eden Research
|
Posted - 2009.03.01 15:25:00 -
[110]
Originally by: Nick Turing
Edit: System.IO.Compression.Deflatestream should do the job, I believe.
You are awesome, that worked perfectly. One less file to distribute! ------
|
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.03.01 18:16:00 -
[111]
Originally by: Khrillian Thanks.
I'm not too familiar with generating indicies. What do you want yours to be beyond a simple weighted average? Is most of the problem cleaning up the data before you average it and weighting multi-region indices?
The main problem is that we have to pull a lot of data into the web process; we're talking around 30,000 market orders, plus a goodly chunk of historic price and movement data. We're then taking out all the crap by doing some quartile filtering (aka boxplot outlier filtering), then calculating standard deviation/variance for the remaining orders and filtering out again based on that. It's a very reliable method and does very well, and is actually very fast to do; however, trying to analyse a set of 33,000 items that way is going to result in some problems on any system.
Alternatives are doing daily generations asynchronously but we'd lose the ability to combine regional indexes unless we just did a simple mean on the result, or doing it asynchronously on load on a background worker then spitting it up once it's loaded.
Nearly all of the market is now cached as fragments; my next job is to work out when to expire those caches and to implement that, and then to get heavy-handed on the SQL optimisation and look at other options for order filtering (prefiltering as orders are uploaded is one thing I'm looking at, as it'd reduce most ops to a simple mean calculation). -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Zeerover
Caldari DeadSpace Exploration and Investigations
|
Posted - 2009.03.01 20:13:00 -
[112]
Nice work, a bit laggy but otherwise great.
|
vvizard NOR
|
Posted - 2009.03.02 05:11:00 -
[113]
This sounds incredible interesting. I cant at first glance understand why this should be EULA-illegal (without checking it). These files are on my HDD, if they cant be used to get a market advantage, they shouldn't be on the HDD in the first place if you ask me.
But tbh, until you get a developer API, you're no good to me. My tools rely on the developer API from eve-central, not their web frontend.
|
ksc1226
|
Posted - 2009.03.02 06:12:00 -
[114]
Originally by: vvizard NOR This sounds incredible interesting. I cant at first glance understand why this should be EULA-illegal (without checking it). These files are on my HDD, if they cant be used to get a market advantage, they shouldn't be on the HDD in the first place if you ask me.
But tbh, until you get a developer API, you're no good to me. My tools rely on the developer API from eve-central, not their web frontend.
First of all thats terrible logic. Eve is on my computer...that doesn't mean I can go around editing all of their files and messing with the program.
Also they do have an API. http://www.eve-metrics.com/developers/help
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.03.03 00:06:00 -
[115]
Originally by: vvizard NOR This sounds incredible interesting. I cant at first glance understand why this should be EULA-illegal (without checking it). These files are on my HDD, if they cant be used to get a market advantage, they shouldn't be on the HDD in the first place if you ask me.
But tbh, until you get a developer API, you're no good to me. My tools rely on the developer API from eve-central, not their web frontend.
It's not our place to question the EULA :) we abide by it just fine, in any case.
The developer API exists and has done since day one. Press 'developers' in the main menu for more information on that, as the post above describes.
We've just finished sorting out the write-through cache which has improved performance dramatically and helped server load too. I'm in the middle of sorting out some major(ish) changes to the way we do order filtering which will reduce load a huge amount and let us bring the indexes section back with any luck! -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
Khrillian
Minmatar Sebiestor tribe
|
Posted - 2009.03.03 01:56:00 -
[116]
Is posting source to a cache reader actually against the EULA, even though it evidently isn't against the EULA to use one? Are you just doing it as a favor to CCP?
|
Capt Nightmare
Gallente Eve Logic
|
Posted - 2009.03.03 04:38:00 -
[117]
This is nonsense!
Keep your mumbo jumbo hocus pocus trading tools away I tell you! You will be the death of trading!
Burn them all!!!!
|
Ahro Thariori
|
Posted - 2009.03.03 12:32:00 -
[118]
This is a truly great project and it's always nice to see someone spending their spare time to make all our lifes easier. I even think that the objections ala "you guys get the data first", "you can take over my business" are invalid. With the kind of time sunk in a project like this, Ix&Co would be much better off putting that time into RL jobs and then converting that RL cash into isk via GTCs. I am positive that there is more to this than making Ix an isk billionaire.
Now, I have some questions, too:
1) Who is on the team besides Ix Forres, Salvis Tallan and "kind of" YunFu Yan? How open is it? Could I contribute later this year when I have more spare time? Will this site die if you suddenly decide to leave EVE in half a year? I am asking because I understand that e.g. eve-central is more than a one-and-a-half-man show (maybe I am wrong).
2) Had you considered teaming up with eve-central? Before you got this far ofc. It seems your most important contribution is the cache reading. Building your own site is probably just an exercise for you. Is it? Is it a case of "not-invented-here" ;)? Or do you have some long-term interest in exercising control over the whole process ? Eve-Central seems to have stable servers and a stable API, unlike yourself. If you care to elaborate: what is your strategic goal in superseeding eve-central?
3) Will the dev api always be free and open (even once you introduce paid-for services)?
Thx,
Ahro |
Hexxx
Minmatar
|
Posted - 2009.03.03 12:38:00 -
[119]
Originally by: Ahro Thariori
2) Had you considered teaming up with eve-central? Before you got this far ofc. It seems your most important contribution is the cache reading. Building your own site is probably just an exercise for you. Is it? Is it a case of "not-invented-here" ;)? Or do you have some long-term interest in exercising control over the whole process ? Eve-Central seems to have stable servers and a stable API, unlike yourself. If you care to elaborate: what is your strategic goal in superseeding eve-central?
They should feel free to develop independently, that's the nature of competition.
My feelings on Banks are similar; competition, new and different services, etc is good for the markets.
EBANK - Chairman of the Board | www.eve-bank.net
|
Ahro Thariori
|
Posted - 2009.03.03 12:50:00 -
[120]
Originally by: Hexxx
Originally by: Ahro Thariori
2) Had you considered teaming up with eve-central? Before you got this far ofc. It seems your most important contribution is the cache reading. Building your own site is probably just an exercise for you. Is it? Is it a case of "not-invented-here" ;)? Or do you have some long-term interest in exercising control over the whole process ? Eve-Central seems to have stable servers and a stable API, unlike yourself. If you care to elaborate: what is your strategic goal in superseeding eve-central?
They should feel free to develop independently, that's the nature of competition.
My feelings on Banks are similar; competition, new and different services, etc is good for the markets.
I don't know if that is a valid comparison. There is a lot to compete about with banks. Or vice-versa: if your comment is dead-on then there is more to his project than community spirit. Which ofc isn't bad per se.
|
|
|
|
|
Pages: 1 2 3 [4] 5 6 7 8 :: one page |
First page | Previous page | Next page | Last page |