Pages: 1 2 3 4 5 6 7 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 14 post(s) |
|
CCP Fallout
|
Posted - 2011.08.30 14:06:00 -
[1]
Attention market addict! Dr.EyjoG's newest dev blog gives us information about a new initiative: providing historic market data to capsuleers. This is a test run, so please do share your feedback with both Research & Statistics and Team Sleeper Cell in this thread.
Fallout Associate Community Manager CCP Hf, EVE Online Contact us |
|
Gnulpie
Minmatar Miner Tech
|
Posted - 2011.08.30 14:08:00 -
[2]
Just awesome! |
Squizz Caphinator
Woopatang
|
Posted - 2011.08.30 14:09:00 -
[3]
yes! yes! yes! -- Eve Who EveChatter KillWhore Killboard |
Femaref
Armageddon Day WE FORM VOLTRON
|
Posted - 2011.08.30 14:09:00 -
[4]
omg, this is epic.
From the two, I prefer csv of course, but others would be better.
|
Lost Hamster
Hamster Holding Corp
|
Posted - 2011.08.30 14:11:00 -
[5]
Nice one. Please keep the SQL version. I think that's the easiest to use. -------------------------------------------------------------------------------------------- Shields are like pants, they're supposed to come off. Armor is like the condom once its gone ur ****ed |
Mors Magne
Astral Adventure
|
Posted - 2011.08.30 14:12:00 -
[6]
Quote: "Fly safe but remember to destroy stuff - it is good for the EVE economy!"
|
Femaref
Armageddon Day WE FORM VOLTRON
|
Posted - 2011.08.30 14:16:00 -
[7]
Originally by: Lost Hamster Nice one. Please keep the SQL version. I think that's the easiest to use.
It's not sql, it's mssql backup. Put another way: PITA.
|
Myxx
Atropos Group
|
Posted - 2011.08.30 14:17:00 -
[8]
I prefer Csv.
is it possible to keep both for those of us who like one but not the other? --
Originally by: CCP Explorer (and if you guys would also stop using Drakes it would be really appreciated, kthxbye).
Originally by: Tom Gerard
Then again... I am a moron.
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 14:21:00 -
[9]
Thank you for this data. It's very useful. So useful, in fact, that we already collect it :o)
When you view the market history in-game, the data is written to a cache file. As cache reading is legal, we have a program that reads the cache file and uploads that data to our server. We also have a JavaScript tool for the IGB that opens the market window for a number of "interesting" types. This used to be provided by eve-metrics, but they closed down. :-( We have most of the data from eve-metrics, with some item histories going back as far as May 2008.
What we use it for
We generate a 7-day "index" price for materials in the regions we operate in. We buy materials below that index, calculate our production cost based on that index, and sell products above that calculated production cost. This allows us to easily cooperate between multiple players (some buy, some sell) without complicated record keeping.
The resulting index is public and can be found here: http://www.electusmatari.com/gmi/
What format would be useful?
Well, it ends up in an SQL table here. Getting it in SQL directly is good. But to translate .bak files into SQL requires some MS program, and for the official DB dump, we actually rely on a 3rd party (zofu for president) to translate it to PostgreSQL format. I suspect a lot of others use MySQL. MSSQL backup format is pretty much unusable for most people, and CSV is easier to translate into something sensible.
Have you considered providing an API interface instead of file dumps?
|
|
Chribba
Otherworld Enterprises Otherworld Empire
|
Posted - 2011.08.30 14:22:00 -
[10]
There we go.
Secure 3rd party service | in-game 'Holy Veldspar' Now /w voice |
|
|
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.08.30 14:26:00 -
[11]
I think this is a great initiative. I hope the developers are also looking at services like eve-central which are providing this sort of information already :)
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
rofflesausage
|
Posted - 2011.08.30 14:32:00 -
[12]
Edited by: rofflesausage on 30/08/2011 14:32:34 Out of the two formats listed, CSV.
.bak files are a pain to work with in my experience.
|
Rees Noturana
Red Rock Mining Company
|
Posted - 2011.08.30 14:36:00 -
[13]
CSV means simpler access to a lot of different programming languages.
MSSQL backup format means evil MS land, angry, angry, flame war. I think we've had enough of those lately. I don't need the added hassle of converting the data into something simple and readable before I run my script on it. Open standards beats weird MS garbage any day.
|
Meissa Anunthiel
|
Posted - 2011.08.30 14:39:00 -
[14]
Wonderful,
This has been a long time coming but is very very welcome. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Jareck Hunter
Rubicon Legion
|
Posted - 2011.08.30 14:47:00 -
[15]
About the Delay. I think i'm not only speaking for myself here, but most of the time, i'm not interrested in the price of tritanium 3 months or 2 years ago, if i want to feed my sheets for production and other things, i want to know how much something cost today, maybe yesterday or last week, so i can work with those numbers.
I think sites like eve-metrics or eve-central, got used a lot cause they provided actuall data, that could be read out automaticly.
So an update every downtime for items that get traded a lot, every 2-3 days things that get traded sometimes and maybe every week for low transaction items could be a way to go.
If you could archive something like that, that would be nice. ------------------------------------------------- Sorry for my bad english^^ |
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.30 14:50:00 -
[16]
Any text format (CSV, JSON, XML, etc..) over a MSSQL backup. If you go with MSSQL backup we have to rely on someone with a copy of MSSQL to do the conversion for us. Seeing as you already have it please do it for us. Thanks!
|
Face ofBoe
|
Posted - 2011.08.30 15:01:00 -
[17]
Anyone who can restore an mssql bak can just as easily consume a csv. Just provide the csv and we can take it from there.
|
Mechael
Minmatar Helhest 1st Prospectors' Collective
|
Posted - 2011.08.30 15:04:00 -
[18]
.csv format would be greatly preferred. MSSQL is a pain in the ass. I'd like to request JSON as I tend to code in node.js and mongodb, but that wasn't an option.
As to a reasonable interval, I'd like to request as often as possible. An API would be preferred, actually. Please, don't make us wait longer than one week between updates.
This will be a huge, huge help and very powerful tool. With what tools we players currently have, in order to get complete market data we have to have someone (or multiple people) manually look up every single item in every single region so that we can read the cache files. It's not very feasible, and very obviously a large headache.
|
|
CCP Stillman
|
Posted - 2011.08.30 15:08:00 -
[19]
Originally by: Dierdra Vaal
As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?
Originally by: Arkady Sadik
Have you considered providing an API interface instead of file dumps?
It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.
|
|
Avall
|
Posted - 2011.08.30 15:11:00 -
[20]
This is awesome!
1) Any text format is preferable to .bak. Personally I prefer XML.
2) Please please please make this an API call, per item. This way it could be used in web apps or mobile apps live, without having to download a large file or a lot of data.
3) Delay should be in the range of hours, absolute maximum is one day. Otherwise the data won't be used very much since more up to date data is available from crowd sourced cache reading repositories like eve central and others.
Looking forward to integrating this into Leaky Capsule, my iPhone app (yes, that was a shameless plug :)
|
|
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.08.30 15:16:00 -
[21]
Edited by: Dierdra Vaal on 30/08/2011 15:15:54
Originally by: CCP Stillman
Originally by: Dierdra Vaal
As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?
Originally by: Arkady Sadik
Have you considered providing an API interface instead of file dumps?
It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.
If you do I'll buy you so much beer at fanfest <3
For my project the eve-central service is sufficient. Min, Max, Avg and Median prices per item, while allowing filtering per region or time period and allowing me to query multiple itemIDs at once.
The biggest thing I'm missing is similar price statistics for faction and deadspace items (basically anything that cannot be traded on the market), but I suspect thats a whole different set of problems.
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
Vanessa Vansen
|
Posted - 2011.08.30 15:21:00 -
[22]
Edited by: Vanessa Vansen on 30/08/2011 15:21:35 About the format ... please use the most general format, i.e. CSV over MSSQL backup
however, I would vote for xml
|
|
CCP Stillman
|
Posted - 2011.08.30 15:23:00 -
[23]
Originally by: Avall Delay should be in the range of hours, absolute maximum is one day. Otherwise the data won't be used very much since more up to date data is available from crowd sourced cache reading repositories like eve central and others.
I just want to address this one already.
From a technical point of view, this is completely impossible I'm afraid. The amount of data it takes to generate this sort of data is absolutely huge. There has to be a delay, because of that. And it has to be measured in days.
From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
|
|
Sgt Napalm
Veto. Veto Corp
|
Posted - 2011.08.30 15:23:00 -
[24]
How much will the subscription based license cost in Aurum?
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 15:24:00 -
[25]
Well, as mentioned, I think the most interesting data is roughly the last 28 days at most. There are uses for older data, but they're rare. For example, I have used old market data in the past to figure out which patch introduced a change that caused a significant price changes (there's a nice bulge in Tritanium which lasted for a year and starts and ends at specific patches, say).
That is, most applications will be interested only in the last few days of data. Our own application would do a daily query of about 380 types (if this is API-queryable, maybe even all types with a marketgroupid) once per day to get the last 7 days of data for four regions, nothing else.
|
|
CCP Stillman
|
Posted - 2011.08.30 15:24:00 -
[26]
Originally by: Dierdra Vaal
The biggest thing I'm missing is similar price statistics for faction and deadspace items (basically anything that cannot be traded on the market), but I suspect thats a whole different set of problems.
I'm afraid that's outside the scope of this project, and is a very difficult problem in general. Sorry.
|
|
Eve Industrialist
|
Posted - 2011.08.30 15:25:00 -
[27]
Looking forward to this. Api similar in form to eve cental xml would be best. Csv is doable as well. My first instict was to say sql but then I realized I am not likely to keep it up to date as such. A delay or csv only release would keep the eve central guys in business... tho I agree a delay longer than a day on mineral prices makes this data not as good as eve central.
|
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.08.30 15:25:00 -
[28]
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
Given how quickly markets can change, I think there will be many cases where 7 days will be too long a delay. Could it be a daily updated thing?
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
Nahkep Narmelion
Gallente CALIMA COLLABORATIVE Atrox Urbanis Respublique Abundatia
|
Posted - 2011.08.30 15:26:00 -
[29]
CSV is preferred, but the file is too big for Excel. I'll likely use it in SAS for analysis, and csv files can be directly exported into SAS. But my guess is not many people will have access to something as powerful as SAS, so if the data could be broken up it might be even more useful.
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 15:27:00 -
[30]
Stop posting while I post! <.<
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
For our uses, not useful at all. We can get up-to-date data with our existing toolchain, and we use the data from the last 7 days. Anything before that is interesting mostly for historical reasons, or for items that are traded very rarely.
|
|
Eve Industrialist
|
Posted - 2011.08.30 15:34:00 -
[31]
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
I would gladly use this... when eve central returns a host not found error or a zero price.. this would be my fall back data source. In fact I now cache eve central prices for this reason and warn users when they are old. Needed for odd items in non trade regions. I wil also default to this price when cached price from eve central is too old.
|
Vessper
Indicium Technologies Hephaestus Forge Alliance
|
Posted - 2011.08.30 15:34:00 -
[32]
Almost 6.5 million rows of data - I'll take that in database format. That said, it literally took me 2 minutes to export this from MSSQL to CSV format so I see no reason why CCP can't easily provide both to satisfy everyone.
As for timing, the more recent the better.
EveHQ Character App |
GizzyBoy
|
Posted - 2011.08.30 15:35:00 -
[33]
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
I just spent 3 LONG nights integrating 2 different data streams for current and pricing semi current pricing, and then you spit this out @#$##$
and that note, it looks like I'll be adding the next one in the next few days,
can we get some kind feed from eve? im just after avrg's & volume traded atm.
|
GizzyBoy
|
Posted - 2011.08.30 15:38:00 -
[34]
FYI just zip the csv,
mssql is a pain, because each ones a diff format...
tho tables are nice, but if you flat file it it should be ok. i guess you could do monthly or weekly packs? maybe even once a night 1/2 hour after dt for previous day?
|
Yohai Hakachi
|
Posted - 2011.08.30 15:39:00 -
[35]
Before offering my .02 ISK on this, I would like to congratulate the CCP team, and Dr. Eyj=lfur "Eyjo" Gu=mundsson in particular for their initiative to make historic market transaction data available.
Formatwise, I am very much in favour of CSV, for the simple reason that it can be imported in just about any program that I know of/use, so these records could full well (and probably will if permission is obtained) be used during statistics/econometrics/economics course(s) etc... for illustration of various topics.
I do have one question however. Will only the data of individual transactions be made available, or will there also be a "daily index", with the average, median, max and min transaction price (on day basis) for the particular items in specific regions?
Yours kindly. |
|
CCP Stillman
|
Posted - 2011.08.30 15:43:00 -
[36]
Originally by: Yohai Hakachi
I do have one question however. Will only the data of individual transactions be made available, or will there also be a "daily index", with the average, median, max and min transaction price (on day basis) for the particular items in specific regions?
The data provided is basically the transaction count, total value, volume, min, max, median and average price based on: - regionID (Top 5) - date (1 years worth in the zip file) - typeID (All types sold on the public market)
We won't be providing individual transaction, just the aggregated information contained in the files in the dev blog
|
|
Meissa Anunthiel
|
Posted - 2011.08.30 15:45:00 -
[37]
Edited by: Meissa Anunthiel on 30/08/2011 15:45:46
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
Quite useful, except for traders.
But then again the more useful the data is for that group of people, the more of a competitive advantage they have over casual traders, and the easier it is to automate (ie, bot).
Anything less than 3 days trailing would be detrimental to casual trading.
And definitely don't provide individual transactions.
That said, 7 days is more than enough for inventors to get a trend on their stock values and whether they should invent something or not, it's also useful for killboards. If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Yohai Hakachi
|
Posted - 2011.08.30 15:50:00 -
[38]
Originally by: CCP Stillman
The data provided is basically the transaction count, total value, volume, min, max, median and average price based on: - regionID (Top 5) - date (1 years worth in the zip file) - typeID (All types sold on the public market)
We won't be providing individual transaction, just the aggregated information contained in the files in the dev blog
Ahh, my mistake. Thank you for clearing that up. |
malaire
|
Posted - 2011.08.30 15:55:00 -
[39]
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
I currently check 1-3 month history when deciding what items to trade in and this could help a lot to find good deals automatically (even if this only contains portion of needed data.)
However, as a programmer I could probably get this data myself quite easily using cache readers, so I don't really mind if you decide to drop this idea completely. (Less information for my competitors ...)
|
Rakshasa Taisab
Caldari Sane Industries Inc.
|
Posted - 2011.08.30 15:56:00 -
[40]
Originally by: Meissa Anunthiel That said, 7 days is more than enough for inventors to get a trend on their stock values and whether they should invent something or not, it's also useful for killboards. If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs.
If you want the trade hub price, you use the median price.
|
|
Rees Noturana
Red Rock Mining Company
|
Posted - 2011.08.30 15:59:00 -
[41]
@CCP Stillman
What is different about the way third party services process market data compared to what you are thinking about doing that requires a delay of days to get at the data? Perhaps a different approach can be taken to streamline price computation.
For an API I'd like what eve-central.com provides but from the official API server. I don't need a historical dump of the last year. Maybe you can provide a regular canned data dump for the historical data. Just publish that somewhere on a regular basis for those that want to generate charts and do more analysis.
|
stoicfaux
Gallente
|
Posted - 2011.08.30 15:59:00 -
[42]
Originally by: CCP Stillman
From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.
People already have tools to get mostly up to date and/or historical information, so the "huge advantage" already exists. Making the "huge advantage" a standard feature would just let more people in on it.
Quote: So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
I think Arkady Sadik has the answer in post #30.
----- CCP's NeX Pricing Tiers Affordable: One PLEX Mid: 3-4 PLEX Deluxe: Only for "flamboyantly rich capsuleers" Exceptional: ?? |
Wind Jammer
Minmatar Molden Heath Software Company
|
Posted - 2011.08.30 16:14:00 -
[43]
Originally by: CCP Stillman
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
Hello CCP Stillman,
I was asking for something along these lines at one of the Dev Track roundtables at Fanfest, so I'm very happy after reading the dev blog and forum posts. I'm the developer of EVE Production Mixer and a few other market analysis tools that aren't public.
I think there are two angles that people will come at this when looking for price data.
Firstly, (this is the case for me and EVE Production Mixer), I'm interested in broad averages to get realistic prices that reflect what a user can actually achieve on the market.
The other angle is people who are looking for 'special sale prices' or unique one-offs that are way outside the averages. To find profitable deals like this I guess you'd need data updates by the minute.
At the moment, I refresh price data from Eve-Central daily. I don't want to diss Eve-Central because they do a great job, but my main problem with Eve Central data is that very low volume items often don't have a price.
In your question about frequency and delay, I would like daily data, and the for the previous day (may be 2 days previous is ok). A good example why: earlier this week someone bought up the entire stock of Fernite Carbide from Jita. For a day, the price was 20% higher than it had. Thanks to the daily price updates, EVE Production Mixer showed a realistic price the next day. I wouldn't be so keen to see that price spike appear in the data a week or 2 weeks later.
Anyway, keep up the good work CCP )
|
Amsterdam Conversations
|
Posted - 2011.08.30 16:20:00 -
[44]
Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.
|
Niraia
Seekers of a Silent Paradise
|
Posted - 2011.08.30 16:27:00 -
[45]
Cool stuff. MSSQL backup is my prefered format *hides* - gallenteprime.com |
|
CCP Stillman
|
Posted - 2011.08.30 16:33:00 -
[46]
Originally by: Wind Jammer In your question about frequency and delay, I would like daily data, and the for the previous day (may be 2 days previous is ok). A good example why: earlier this week someone bought up the entire stock of Fernite Carbide from Jita. For a day, the price was 20% higher than it had. Thanks to the daily price updates, EVE Production Mixer showed a realistic price the next day. I wouldn't be so keen to see that price spike appear in the data a week or 2 weeks later.
We'll be providing the data on a per-day basis. The question just is, what sort of delay there is on the data :)
|
|
radix29
Unforeseen Consequences. THE UNTHINKABLES
|
Posted - 2011.08.30 16:34:00 -
[47]
MSSQL all the way.. btw you can download and install SQL Express for free.
|
Cypreion
|
Posted - 2011.08.30 16:39:00 -
[48]
I think XML would be the easiest format for this kind of thing no?
|
|
CCP Dr.EyjoG
|
Posted - 2011.08.30 16:42:00 -
[49]
Originally by: Amsterdam Conversations Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.
Ahhh, you saw right through me, didn't you
The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
|
|
Yvan Ratamnim
|
Posted - 2011.08.30 16:46:00 -
[50]
WAIT.. WHAT
too much processing to provide 1 hour delayed data? WTF ARE YOU TALKING ABOUT
Its already available to an extent via sites like eve-central etc, so its already being exported...
It doesnt have to be a 1 hour or 6 hour delayed from when i first grab it, it can be from set eve times, 0:00 6:00 12:00 18:00 dump it and anything in between is just cached responses...
I mean honestly all the data is already being exported by third parties "he hard way" so why not just do it like this and make everyone happy?
|
|
Rees Noturana
Red Rock Mining Company
|
Posted - 2011.08.30 16:57:00 -
[51]
Originally by: radix29 MSSQL all the way.. btw you can download and install SQL Express for free.
Windows isn't the only operating system in the world. Data should be in an open format.
|
Jareck Hunter
Rubicon Legion
|
Posted - 2011.08.30 16:59:00 -
[52]
Mhh what about the idea of providing different dumps with different delays for specific marketgroups.
Minerals maybe get a daiyly one, posparts/moonmins a weekly one or so? ------------------------------------------------- Sorry for my bad english^^ |
Rees Noturana
Red Rock Mining Company
|
Posted - 2011.08.30 17:00:00 -
[53]
I'm thinking there is a difference in perspective going on here. CCP is talking about providing us with historical data, which is nice to have. But, many of us just want a way to query prices for our trade and production models. If we can get near real time median prices from trade hubs and also get batch downloads of historical data I think you could satisfy most people.
Yes, I can live with 24-48 hours for computing production costs.
|
Max Kolonko
Caldari Worm Nation Ash Alliance
|
Posted - 2011.08.30 17:09:00 -
[54]
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
Originally by: CCP Dr.EyjoG As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
I would love for it to be no more than 2-3 days old. I don't need it to be updated to the minute.
For the sake of traders competition I dont like it to be anything else (it would be great, but i gueass it would slowly kill casual, less tech savy traders)
Pls make it in API format so we can send something like: - item ID(or rather list of ID's) - Region - Date (or list of Dates) to limit our queries.
And pls make sure that Your API calls will work nice with Google Docs ImportXML function :)
PS: Great Job Guys Max Kolonko |
Durin Sarga
|
Posted - 2011.08.30 17:11:00 -
[55]
Would love to have an API version of the historical data available on a daily basis, as we can already get from cache readers or see in the historical table provided in the EVE client.
I think daily granularity is enough for inquisitive minds, and a 1 day delay is reasonable.
Just my 2 ISK.
|
Thebriwan
LUX Uls Xystus
|
Posted - 2011.08.30 17:21:00 -
[56]
This is awesome news!
I personally would like to have the data in an XML-format. Second best would be CSV.
I see two different points in this discussion: a) producers are needing actual data for item _for_ production b) traders and statistic freaks need massive data for _as much items as possible_
So I propose two different approaches: a) make all items from the primary producer index available starting from the last downtime b) all others can wait 2 or 3 downtimes I think
|
Deamos
Dev Null Development and Holdings
|
Posted - 2011.08.30 17:22:00 -
[57]
+1 for API of market data -
|
Cypreion
|
Posted - 2011.08.30 17:30:00 -
[58]
XML and JSON are formats of choice sans-whole db dumps.
|
Iam Widdershins
Project Nemesis Moar Tears
|
Posted - 2011.08.30 17:41:00 -
[59]
Edited by: Iam Widdershins on 30/08/2011 17:42:52 I think this is an OK idea, but the people who currently make serious isk form trans-region market trading will detriment from an influx of overly-informed traders using this API.
I wager that not a whole lot of the people asking for almost-live market data in this thread are currently involved in market trading.
Please implement a substantial delay, 7 days or more.
|
Iam Widdershins
Project Nemesis Moar Tears
|
Posted - 2011.08.30 17:45:00 -
[60]
Originally by: Meissa Anunthiel If you're going for 7 days, I'd add systemID as well, because the average price in a region is less important to most than the average price in a system, specifically trade hubs.
This would be TMI. Look at the data dump as it is now, then imagine that 70x bigger.
|
|
Lutz Major
|
Posted - 2011.08.30 17:47:00 -
[61]
Originally by: CCP Dr.EyjoG ... [snip] ...
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
24-48 hours would be awsome!
Stillman: A API call would be nice, but wouldn't the sheer amount of requests by - uhm - everybody (!) have an impact on the QoS of the current API server? Not everybody sticks to the cachedUntil times
I think a zipped CSV is best use for all operating systems.
|
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.08.30 17:56:00 -
[62]
Originally by: CCP Dr.EyjoG Does anyone see a problem with providing 24 - 48 hour old data?
for my purpose, no :)
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
|
CCP Stillman
|
Posted - 2011.08.30 17:57:00 -
[63]
Originally by: Lutz Major
Stillman: A API call would be nice, but wouldn't the sheer amount of requests by - uhm - everybody (!) have an impact on the QoS of the current API server? Not everybody sticks to the cachedUntil times
I think a zipped CSV is best use for all operating systems.
Performance is one of the main concerns we have to work with to deliver this. But there's a lot of caching that can be done. For one thing, all the data will be cached in memcached, and we can easily turn on output caching.
So while performance is definitely a concern, one of the things we won't let happen is degrade the API performance. So I wouldn't worry too much to be honest
|
|
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.30 17:57:00 -
[64]
Edited by: Marcel Devereux on 30/08/2011 18:00:07 Edited by: Marcel Devereux on 30/08/2011 17:59:49 Generating one large dump in one pass is very expensive. But dumping out the history for a individual item by region is not. You already do this and the client can request it. People have written webpages that continually cycle through the market items using javascript and the IGB. They then upload the cache files of both the history and the open orders to various sites. You can do this request and put the information in flat files and let us access those. This would probably reduce database load as people would stop constantly updating orders in the client.
Again there is NO reason to create a HUGE dump file. ONE file per ITEM by REGION. I request that I have is that you generate a yearly dump file, again per item by region, so we can have access to all historical data for the items. For example you for 2007 Trit data in The Forge would look like: 2007-34-10000002.csv. You should not have any DB performance problems generating this file. If you do then you have bigger problems that you should be working on ;-)
I would also like to see the number of buy/sell transactions.
|
malaire
|
Posted - 2011.08.30 18:20:00 -
[65]
Originally by: CCP Dr.EyjoG The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
I oppose the idea of giving too fresh information too easily. That would take advantage off from players who can use cache readers to implement this by themselves ...
So. 7+ days is acceptable, 1-7 days is worse and less than 24 hours is just awfull.
|
Salpun
Gallente Paramount Commerce
|
Posted - 2011.08.30 18:23:00 -
[66]
I am not a market junkie but if you are recalculating the numbers already and do add the buy sell numbers add a third data set identifing the number of buy sell orders that are outside 40% ? of the mediam price. So analysts have a reliability indicator built into the system. So outlier buy/ sell orders can be IDed as a significant factor in the price or if one centing was the prime mover. Just a thought.
|
malaire
|
Posted - 2011.08.30 18:24:00 -
[67]
Originally by: Marcel Devereux I would also like to see the number of buy/sell transactions.
There is already transactionCount. If you mean separate counts for buy and sell transactions, then perhaps you forgot that each transaction is both buy and sell, depending on whether you ask from buyer or seller.
|
mini meee
|
Posted - 2011.08.30 18:29:00 -
[68]
Originally by: CCP Dr.EyjoG
The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
Does anyone see a problem with providing 24 - 48 hour old data?
With the existence of the browser api, jscript and eve-metrics old uploader - it's basically possible to get the market history data automatically at the moment from the cache [up to previous day] - This is what eve-metrics used to do, however their code for doing that effectively took 4-5 hours to loop through items.
Similarly in game, we are currently used to the history in game showing a daily min/avg/max.
Therefore *ideally* accurate data, once per day at midnight would provide what it s currently possible at the moment.
So Suggestion: how about a weekly/monthly csv containing all history, and then a daily 'update csv' file processed at midnight each day that contains the previous days data for min/avg/max. [I'm assuming that those would be low maintainence as I assume you don't calculate the ingame history data on the fly :)]
I'm also assuming that <24 hours data isn't that useful - if I'm looking to buy 200mil trit, and it's 3isk in Jita on current market orders. I like to know what the price was yesterday i.e. has price gone up/down, is todays price 'fair' - for that the previous day would be useful - prices can change a lot in a week.
|
Gallosek
|
Posted - 2011.08.30 18:31:00 -
[69]
Edited by: Gallosek on 30/08/2011 18:32:50 never mind :)
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.30 18:33:00 -
[70]
Edited by: Callean Drevus on 30/08/2011 18:35:45
Originally by: malaire
Originally by: Marcel Devereux I would also like to see the number of buy/sell transactions.
There is already transactionCount. If you mean separate counts for buy and sell transactions, then perhaps you forgot that each transaction is both buy and sell, depending on whether you ask from buyer or seller.
Though it's pretty useful to know whether it's a buy or a sell order being fulfilled :) on a different note, since when are you in favour of cache scraping?
In any case, this is completely awesome! I wouldn't mind any delay less than a week or so.
It means I'll always have the historical data available, and people will still be interested in the true orders in a region, so services like EVE Marketeer are not immediately defunct (but won't have to gather the annoying history tab information anymore, meaning more focus on true orders).
As for the format, CSV is most desired.
Data would have to be available in episodes. Probably per day or per month. The person above me describes it well.
UPDATE: As for the timeline on the data, how far back will this data go? Personally I'd be very interested in seeing the history of prices even back to the launch of EVE, even if they aren't useful for my current trading (but I'm a data freak). ---
|
|
Haguu
Caldari TLA Ltd
|
Posted - 2011.08.30 18:37:00 -
[71]
As many, many people have said no Microsoft formats at all. ever. After that my initial reaction is a tossup to CSV or XML with JSON a somewhat distant third.
Until you are prepared to go [near] real-time, why bother with an API that people have to code cache timers for. If you had a static file name like EY20110830.zip then we could download the static file without your database engine being effected by performance. If you really cared, then put the files in Google files or EVE files where the downloads have zero impact on CCP bandwidth. Don't go to the expense of an API transaction if we are not benefitting from it. Similarly, it could easily be that a single static file is easier to compute and serve up than a lot of players and site requesting hundreds of queries. Plus a single static file maintains a history so if some player wanted to audit what was that price on that past day, it is available. Which an API query of a database might not still have available.
I do see some uses for true historical data (trends of trit % of BS market basket over last few years or plex price changes prior to patches)
What I think would be awesome and yet not that expensive to do is a nightly dump of all items in Jita 4-4 (in addition to a whole Universe every 3-7 days)!!!! Worst case, the one from the prior downtime? Or release the previous DT data 12 hours after DT (so during DT you just copy and you have 12 hours afterwards to run the data.)
This is a much smaller dataset to compute; much smaller to serve up. (Again, make sure it is a static file not some API overhead.) This is all the killboard sort of applications really need. It does not impact the real-time trader searching the universe for arbitrage. It is probably good enough for the lower end "what can I manufacture?" "what ore should I mine" sites? Even if a manufacturer has special arrangements for their costs or sales, Jita is the "list price." There are many times when people want to know the reference price.
So please: a frequent static zipped CSV file of just Jita 4-4 prices???
|
malaire
|
Posted - 2011.08.30 18:42:00 -
[72]
Originally by: Callean Drevus
Originally by: malaire
Originally by: Marcel Devereux I would also like to see the number of buy/sell transactions.
There is already transactionCount. If you mean separate counts for buy and sell transactions, then perhaps you forgot that each transaction is both buy and sell, depending on whether you ask from buyer or seller.
Though it's pretty useful to know whether it's a buy or a sell order being fulfilled :) on a different note, since when are you in favour of cache scraping?
True, however in some cases both are true (player making buy/sell order which is instantly fulfilled by existing sell/buy order).
And in my other thread I didn't oppose cache reading as much as IGB javascript automation to update the cache.
|
Salpun
Gallente Paramount Commerce
|
Posted - 2011.08.30 18:53:00 -
[73]
Is created sell orders and there mean price enough data to serve as the sanity check but small and simple enough enough that CCP might add it to the list.
Processed transactions verses outstanding transactions is the issue. Eve-Central pulls outstanding transactions both buy and sell. CCP is wanting to give us Processed transactions.
|
Palovana
Caldari Inner Fire Inc.
|
Posted - 2011.08.30 19:00:00 -
[74]
CSV format, which can be imported into spreadsheets or any (free) database we'd be using.
MSSQL backup format is far less useful to those not using MSSQL.
----- CCP is still under the misconception that by waving a bunch of (NeX store items) in our faces, we'll give in to our urges and (buy) them. Bring back the hangar view and its functionality! |
malaire
|
Posted - 2011.08.30 19:13:00 -
[75]
It would be usefull to have both min/max prices which include outliers, and other min/max prices which does not.
|
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.30 19:15:00 -
[76]
Originally by: malaire
Originally by: Marcel Devereux I would also like to see the number of buy/sell transactions.
There is already transactionCount. If you mean separate counts for buy and sell transactions, then perhaps you forgot that each transaction is both buy and sell, depending on whether you ask from buyer or seller.
Ok Obi-Wan. How about this point-of-view? The counts of how may buy orders and sell orders were fulfilled.
|
Meissa Anunthiel
|
Posted - 2011.08.30 19:17:00 -
[77]
Originally by: CCP Dr.EyjoG
Originally by: Amsterdam Conversations Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.
Ahhh, you saw right through me, didn't you
The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
Makes it more difficult for some traders to profit by making it clearer where the opportunities are (thereby creating competition where there was little or none). Information is power when it comes to markets Eyjo, so the shorter the span, the bigger the damage.
Then again, it may also creates opportunities for enterprising individuals. For markets that have big transaction volume, 24 hours wouldn't be a big deal, for "more remote" regions, it would. Personally I think 3 days is more than enough, it gives enough information that people would have to move their arses and see what it's like "down there" right now.
----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Salpun
Gallente Paramount Commerce
|
Posted - 2011.08.30 19:19:00 -
[78]
Hi Meissa
Glad to see a CSM joining the discussion.
|
Tau Cabalander
Retirement Retreat
|
Posted - 2011.08.30 19:22:00 -
[79]
Edited by: Tau Cabalander on 30/08/2011 19:26:13
CSV is preferable to a proprietary (Microsoft) format. Any open format is preferred.
I'll just import it into open source packages like SQLite, or even MySQL.
If the dump isn't current, expect the API to be hammered as a result.
|
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.30 19:25:00 -
[80]
Originally by: Meissa Anunthiel
Originally by: CCP Dr.EyjoG
Originally by: Amsterdam Conversations Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.
Ahhh, you saw right through me, didn't you
The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
Makes it more difficult for some traders to profit by making it clearer where the opportunities are (thereby creating competition where there was little or none). Information is power when it comes to markets Eyjo, so the shorter the span, the bigger the damage.
Then again, it may also creates opportunities for enterprising individuals. For markets that have big transaction volume, 24 hours wouldn't be a big deal, for "more remote" regions, it would. Personally I think 3 days is more than enough, it gives enough information that people would have to move their arses and see what it's like "down there" right now.
And by bigger the damage you mean more tears. This is EVE. They will need to HTFU or do something else. Universal access to data and competition is good.
|
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 19:27:00 -
[81]
Originally by: CCP Dr.EyjoG Does anyone see a problem with providing 24 - 48 hour old data?
Depends on what you mean, exactly. Data of the "current" day is available, albeit not very useful ("so far today there have been X items traded at price Y on average"). Bargain traders want market orders, not the market history.
The interesting data are the finalized values per day.
So, let's say we have the 10th of a month (sometime during noon, say, DT). Data of the 10th is theoretically available, but not too helpful. The interesting data starts with the finalized values of the 9th, which is available in-game starting at 00:00 on the 10th. Having that data available out-of-game would be fabolous. Having data from the 8th only would be ok still. Going back further from there would get worse quickly.
To give you an idea, for our current index, we consider data older than 3 days "old", and anything older than 7 days "outdated". (Excluding rarely-traded items, of course.)
The data quantities involved here are not particularly huge. The data itself is already available on the TQ DB and accessible to any client. So the question is transfer volume. For a single day, we're talking about 6844 market types in 67 regions with a market (assuming CCP trades stuff in their jovian regions). The export contains 10 columns, assuming 64 bit each that would be under 40MB of data per day. I do not think that transferring that to a different server once per day would be a particularly huge drain on the database.
It would even be perfectly fine to get text files with the daily data on some file server. cdnsomething/markethistory/20110801.xml.gz with the full data for that day only would be perfectly ok - almost all people using this data will query daily and store it locally anyhow. Even assuming some wasteful XML encoding, this would still be under 100MB for a single day, uncompressed (assuming 20 bytes per number + separators gives 92MB).
Regarding "unfairness" - giving an in-game advantage to people who happen to know how to read cache files sounds like a bad game design choice. It's enough to give an in-game advantage to those who know how to massage data into something useful already, no need to make it more complicated than that.
|
Matthew
Caldari BloodStar Technologies
|
Posted - 2011.08.30 19:40:00 -
[82]
This is pretty awesome.
In terms of format, .bak is fine for me. Alternatively XML would be preferred and would have the advantage of being able to be unified between static historic dumps and any potential API service for near-live data.
I would suggest that the API and static dumps are deployed as an integrated system. The API can be used to serve the most recent data (maybe last 30 days or so, sufficient to overlap with the static data cycle), with full historic series (back to the beginning of time if possible) provided in static downloads.
In terms of timeliness, 24-48 hour delay on the API calls sounds good. Any longer than that and cache-sc****rs are going to remain the most popular form of acquiring this data.
API Calls
MarketDailyUpdate - Accepts a date as parameter and returns an all-items all-regions dump for that single day.
MarketItemUpdate - Accepts a typeID as parameter and returns an all-regions all-days dump.
MarketQuery - Accepts date, typeID and regionID to allow small specific pulls.
Static Dumps
I suggest a system of files that group-up over time.
Where a full year of data is available, the file is provided by year. For the incomplete year, complete months are provided by month. Potentially then a weekly file for incomplete months depending on the API/static balance that is settled on.
That way new users boot-strapping their records have a reasonable number of historic files to grab, while consumers of ongoing updates don't have large amounts of repeat data to extract.
Data Content
Everything in there looks good, the only thing that seems to be missing compared to the info in the client are the 5 and 20 day averages. While the means can be calculated by the user, the medians cannot be derived locally, so if these could be added that would be good. ------- There is no magic Wand of Fixing, and it is not powered by forum whines. |
Chruker
|
Posted - 2011.08.30 20:08:00 -
[83]
Could use a ticker thingy which would show new orders as they are put up :-) ----- http://games.chruker.dk/eve_online ----- Top wishes: - No daily downtime - Faster training on sisi
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 20:14:00 -
[84]
Minor point: You might want to rename the stationID column to regionID in your csv file. :-P
|
Hel O'Ween
Men On A Mission EVE Trade Consortium
|
Posted - 2011.08.30 20:30:00 -
[85]
Please, please do not release the data in CSV format. CSV has no strong data types nor is it locale aware. It's an ugly hack, the lowest common denominator ... from a couple of decades ago.
Anything is better than CSV. If you're looking for a text-based format, go for XML. 3rd party devs working with the (official) SDE already do have MSSQL installed somewhere. If not - pick up your free copy at MS's website. -- EVEWalletAware - an offline wallet manager |
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 20:33:00 -
[86]
Edited by: Arkady Sadik on 30/08/2011 20:34:58
Originally by: Hel O'Ween 3rd party devs working with the (official) SDE already do have MSSQL installed somewhere.
No, we don't. We wait for zofu to convert it to something we can actually use.
Quote: If not - pick up your free copy at MS's website.
And then we have to still convert the SQL to something the actual databases we use can parse. Which requires quite some regex gymnastics.
Parsing CSV (especially as this dump primarily contains numbers only, no complex strings) is trivial and fault-free in comparison.
XML is ok (better). Just no $/!"& MSSQL .BAK file.
|
E man Industries
|
Posted - 2011.08.30 21:00:00 -
[87]
as a dumb user I like this. Fitting tools like EFT use to calculate how much a given ship would cost. this was very helpfull in fitting. Does that meta 4 items cost a stupid amount? ext.
Also having access out of game to jita prices I can make quick desisions on to sell localy or transport to jita for sale as I can compare the cost.
I do not want the data to be live however and the data should be 1+week old. Let in game speculation rule rather than a data base that is much easier to run through a bot to show diffrences in price. ______ Hello WoW players. Look at your toon, now back to me. Sadly it isn't me, but if it wasn't simplistic pre scripted linear mono dimensional game you could look like me. I'm in a Paladin |
|
CCP Stillman
|
Posted - 2011.08.30 21:08:00 -
[88]
First of all, thanks to everybody who's shared their opinions and comments so far. Keep them coming, they're immensely helpful.
On the format discussion, I want to propose a suggestion. It might be silly, but hear me out:
CSV is a very simple format. I know that at least the database engine I work with on a daily basis(MSSQL), can import a CSV file into a table in about a small line of SQL. And I'd be surprised if other database engines couldn't.
I know for a fact we have players around here who has expert knowledge in other database engines, who would know how how to make a CREATE statement for the data and a query for importing the data into a database.
So how about we throw a page on EVELopedia, where everybody can share how to import a CSV file into their database engine of choice. That means we can release a single format, and if people wish to import it into a database, or any other application one could think of, then they can head on over to EVELopedia and find out how to do it easily.
Does that seem like a nice middle-ground? We can't please everybody, but at least we can make sure that the information for how to consume the information is easily available.
Just an idea. I'd be interested to hear what you all think. -Stillman
|
|
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.30 21:09:00 -
[89]
Originally by: Hel O'Ween Please, please do not release the data in CSV format. CSV has no strong data types nor is it locale aware. It's an ugly hack, the lowest common denominator ... from a couple of decades ago.
Anything is better than CSV. If you're looking for a text-based format, go for XML. 3rd party devs working with the (official) SDE already do have MSSQL installed somewhere. If not - pick up your free copy at MS's website.
Not everyone has Windows. This format needs to be some texted based format.
|
Eybium
|
Posted - 2011.08.30 21:18:00 -
[90]
First off, thank you very much for this information. I am already drawing useful insight from it.
I can confirm Arkady Sadik's observation is true for the SQL backup as well; the stationID field is actually populated with regionID.
Also, although I don't have an opinion on which format is ultimately used for this information, I might suggest that if SQL Backups are the future, to produce those backups in 10.00 or even 9.xx format for users running older instances of SQL Server 2008 and 2005. Currently the dump can only be restored to a SQL Server 2008 R2 instance.
|
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 21:21:00 -
[91]
Originally by: CCP Stillman CSV is a very simple format.
Yes and no. The basic format is trivial, but it gets complicated when you add things like newlines or commas in data sets. Luckily, for this specific data dump, those specialties do not matter at all, so CSV here works exceptionally well. I'm writing this to make sure CSV isn't seen as a great solution for other data exports.
As long as the export only contains numbers and at most dates, CSV works. For anything more complex, use something else.
Quote: Does that seem like a nice middle-ground?
Yes.
Untested, but:
PostgreSQL:
COPY markethistory FROM 'foo.csv' WITH CSV HEADER;
MySQL:
LOAD DATA INFILE 'foo.csv' INTO TABLE markethistory FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES;
SQLite:
.separator "," .import foo.csv markethistory
(There's even an SQLite module to use a CSV file directly as a table...)
|
|
CCP Stillman
|
Posted - 2011.08.30 21:23:00 -
[92]
Edited by: CCP Stillman on 30/08/2011 21:23:53
Originally by: Eybium
I can confirm Arkady Sadik's observation is true for the SQL backup as well; the stationID field is actually populated with regionID.
Ooops. I must have forgotten to change the schema after I worked with the data on a stationID basis as opposed to regionID. My bad
And yes, the stationID field IS the regionID.
|
|
Alain Kinsella
Minmatar
|
Posted - 2011.08.30 21:44:00 -
[93]
Edited by: Alain Kinsella on 30/08/2011 21:47:35
Originally by: Arkady Sadik
Originally by: CCP Stillman CSV is a very simple format.
Yes and no. The basic format is trivial, but it gets complicated when you add things like newlines or commas in data sets. Luckily, for this specific data dump, those specialties do not matter at all, so CSV here works exceptionally well. I'm writing this to make sure CSV isn't seen as a great solution for other data exports.
As long as the export only contains numbers and at most dates, CSV works. For anything more complex, use something else.
The CSV 'format' (of entries separated by commas) is also the default export/import format for Sybase BCP (Bulk-Copy). That handles multiple data types just fine, so long as you don't change the column list over time.
Common command: bcp marketdata in market_file.csv -c -Ufoo -Pfoo -SDB_FOO
I'd probably grab a copy of Sybase SQL Anywhere and use that instead of MSSQL or MySQL, simply due to using ASE and IQ regularly at work (will be adding RAP to that very soon).
@ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).
[Edit - Also, any chance this data dump can be done in OHLC format? (Open-High-Low-Close) Yeah, its a stretch, but I'm curious if you hold that kind of data historically.]
|
E man Industries
|
Posted - 2011.08.30 21:48:00 -
[94]
I really do miss the QER. Was fun to see the ships used and ship counts and other items not contained in this data dump. Will this data still be presented quarterly? ______ Hello WoW players. Look at your toon, now back to me. Sadly it isn't me, but if it wasn't simplistic pre scripted linear mono dimensional game you could look like me. I'm in a Paladin |
Micrurus Decoratus
Caldari
|
Posted - 2011.08.30 21:52:00 -
[95]
Great news.
To be honest I can't see this being too useful to the majority if the delay is greater than 24 hours, as they can always go to eve-central or eve-marketdata for incomplete, but up-to-date data.
However, that's not to say it isn't useful at all. Even with a one week delay, this could prove useful in creating automated market research tools that work with surprising accuracy.
|
Salpun
Gallente Paramount Commerce
|
Posted - 2011.08.30 21:55:00 -
[96]
Originally by: E man Industries I really do miss the QER. Was fun to see the ships used and ship counts and other items not contained in this data dump. Will this data still be presented quarterly?
Lets get this one sorted first I got a feeling we will be seeing alot more data dumps like what we are working on now once a stable system is in place.
|
Snorre Sturlasson
|
Posted - 2011.08.30 22:00:00 -
[97]
Originally by: CCP Stillman
So how about we throw a page on EVELopedia, where everybody can share how to import a CSV file into their database engine of choice. That means we can release a single format, and if people wish to import it into a database, or any other application one could think of, then they can head on over to EVELopedia and find out how to do it easily.
-Stillman
Nice strategy, because a db like postgresql has already a command to import data via CSV.
|
Cearain
Caldari The IMPERIUM of LaZy NATION
|
Posted - 2011.08.30 22:18:00 -
[98]
I'm not much for spreadsheets but I do trade a bit. Mostly by traveling and checking the prices for certain goods in the areas I visit. I suppose any variations where I used to make money will be immediately pounced on by those with spreadsheets pulled up on 12 different monitors.
I'm not sure trading will be in my future anymore.
But anyway, I'm not even sure how to read this. What are the item and system id numbers?
-Cearain
Make fw pvp not pve http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1329906&page=1 |
Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2011.08.30 22:22:00 -
[99]
CSV - It is the lowest common format.
Sandbox Protection League
|
Max Kolonko
Caldari Worm Nation Ash Alliance
|
Posted - 2011.08.30 22:30:00 -
[100]
Originally by: Adunh Slavy CSV - It is the lowest common format.
So is XML Max Kolonko |
|
Rees Noturana
Red Rock Mining Company
|
Posted - 2011.08.30 23:13:00 -
[101]
Originally by: Max Kolonko
Originally by: Adunh Slavy CSV - It is the lowest common format.
So is XML
XML is fine but considering the simplicity of the output, a simple table, having a less verbose format like CSV can really save on downloading. With XML we'll have as much text wrapping the content as we have in content. I'd say CSV is faster to parse as well.
Both the size of the file and speed of parsing really add up with a large data dump.
|
Dr Mercy
|
Posted - 2011.08.30 23:29:00 -
[102]
For me, there is no almost use in the data as it doesn't delineate between market activity from buy orders and sell orders.
Would you be able to provide the same data but give have a pair values for preceded by BUY and SELL to indicate buy or sell orders? |
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.30 23:32:00 -
[103]
Originally by: Rees Noturana With XML we'll have as much text wrapping the content as we have in content.
tr -d '\r' < MarketData.csv | awk -F, 'BEGIN{print "<markethistory>"} NR > 1{print "<history><typeid>" $1 "</typeid><regionid>" $2 "</regionid><datetime>" $3 "</datetime><transactioncount>" $4 "</transactioncount><totalquantity>" $5 "</totalquantity><totalvalue>" $6 "</totalvalue><avgprice>" $7 "</avgprice><maxprice>" $8 "</maxprice><minprice>" $9 "</minprice><medianprice>" $10 "</medianprice></history>"} END{print "</markethistory>"}' > MarketData.xml
571M MarketData.csv 2.1G MarketData.xml
:-D
(This post is meant as humor. If you take it serious, that's your fault. Some more disclaimers: The data is so trivial that you do not need escaping, so the above works. Yes, you could make it a much simpler XML format by using attributes instead of subelements. XML gurus will stalk you at night, though. Compressed, the two files above are 151M (csv) and 203M (xml).)
|
Dr Mercy
|
Posted - 2011.08.30 23:39:00 -
[104]
And as someone on FHC said "To inform the 5-year plan to enhance nullsec and related discussions about self-sufficiency, the data should include ALL regions, not just the top 5 trade regions"
Can you address this point please? |
Jayar Rotuu
|
Posted - 2011.08.31 00:02:00 -
[105]
CSV format please.
|
Agamenox
|
Posted - 2011.08.31 00:24:00 -
[106]
It¦s a terrible mistake give this information to all pilots in game, CCP, must start to evolved and join Roleplaying, PvE and market. Create a new set of missions for industrial, or create a new agents, when the player finish the missions, he could enter to the "secret" market or "smuggler" database with all info.
Stop putting content without doing first a history and game development job.
Eve Mythology and History must grow up!
|
Vaerah Vahrokha
Minmatar Vahrokh Consulting
|
Posted - 2011.08.31 00:27:00 -
[107]
Edited by: Vaerah Vahrokha on 31/08/2011 00:27:09 In order to be of any worth, the data has to provide:
- OHLC data or at least HLC since EvE markets run 24h a day. - Be max 1 day old. - Be of 1 region (selectable), not some aggregate.
Now, I am providing this accurate in game OHLC data service since months:
Thread link on MD
The service provides data (even daily updates), graphs or even an Excel sheet with the data dump.
How can a simple end user be able to produce timely, accurate data (partly based on cache reading) with graphical output but CCP seems to just provide averages and other quite useless (for trading) numbers?
Auditing | Research | 3rd Party | Collateral Holding | EvE RL Charity |
Orisa Medeem
|
Posted - 2011.08.31 01:08:00 -
[108]
Originally by: CCP Stillman
I just want to address this one already.
From a technical point of view, this is completely impossible I'm afraid. The amount of data it takes to generate this sort of data is absolutely huge. There has to be a delay, because of that. And it has to be measured in days.
From a design point of view, there's also the concern about people getting a huge advantage from getting this sort of information.
So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
Guys, please, don't implement a "one size fits all" delay for the information available. There is such a wide range of uses from those 6k item types that no matter which value is chosen some information won't be useful (too old) and some will have a negative impact (too up-to-date information easily available).
I suggest making those delays somewhat dynamic, for example: - Once a month, calculate for each item type and region which delay it should have, using something like C/sqrt(T), C being a constant and T being the total isk traded in the previous month. - fit each result in one of the following intervals [1m, 3w, 2w, 1w, 4d, 3d, 2d, 36h, 24h, 18h, 12h, 9h, 6h]. Default to one week if there is no previous month statistics (for the first run and for new item types created)
This way industrial types will have a handy way to check up-to-date mineral prices while keeping some room for merchant types to search for good deals.
Also, by properly choosing C you can solve the performance problem since only a fraction of the items would have their statistics calculated very often.
|
Monstress
|
Posted - 2011.08.31 01:09:00 -
[109]
CSV would be the most ideal format. Many RDBMS provide a way to import CSV data into tables, for example: LOAD DATA (MySQL) or COPY (Postgres).
|
Anya Ohaya
|
Posted - 2011.08.31 01:39:00 -
[110]
- Definitely CSV format. Not only is it compatible with databases, but also spreadsheets and statistical software like R
- I would love to get longer time series. You cant do proper analysis of seasonal patterns and trading day influences with less than three years of data. It'd be great to put some hard numbers on the effects of holidays etc on prices and volumes
|
|
Abis Cann
Caldari Due North
|
Posted - 2011.08.31 01:39:00 -
[111]
CSV, please. XML being a close second and SQL being a distant third. As to the delay, the shorter the better but I can understand if there needs to be a day or two for data processing.
I'd also like to add that those who worry that a short delay will give an "unfair" advantage to serious traders are absolutely correct that it will be an advantage but I strongly disagree that it would be unfair. If you're willing to learn about markets and economics and put in the time and effort to work with the data then you deserve the advantage. Period. The information will be available to all and it's your decision not to use it so don't complain that others will be less lazy than you and call it unfair. They put in the time, they should reap the rewards.
Moreover, I've heard people claim that a lot of things would kill trading before. The introduction of freighters (and, later, jump freighters in regards to 0.0 markets) was initially feared by some people as the death knell for trading. And, yet, here we are. The fact of the matter is that, unless regional markets approach perfect competition *and* with roughly the same prices server-wide *and* prices never change significantly, trading will be viable. I seriously doubt a daily data dump will create this near-impossible situation, especially since people have been using workarounds for this very purpose for years. Bots are also not an issue if the data is delayed more than 12 hours or so.
Finally, I'd like to say thank you for implementing this. It's been a long time in coming but it's something I've been wishing for since I started trading years ago and is therefore much appreciated.
|
GrayLensman
|
Posted - 2011.08.31 01:48:00 -
[112]
While long overdue I compliment CCP on finally providing an export of near real-time order data. I wrote for my own personal use an application that draws from the data provided by EVE-Central. With a minimal set of alts across targeted regions it is possible to get 24-48 hrs worth of visibility to price differentials to make profit-making between regions for select items possible.
Format is flexible - I use XML against EVE-Central APIs, meaning that while CCP concerns of performance are legitimate - I am in fact self-selecting against a minimal set of item types - which helps with performance (eg. no I am not interested in every type, every region and every price). An export of every type across every region is NOT useful to most pilots - given that - a selective API is the MOST useful application of order exports.
For those pilots without programming skills - some form of CSV export is the most helpful. This format is importable to most users - I leave it to the CCP experts to determine how best to easily export that selectively into Excel, OpenOffice etc.
From a timing perspective - real-time is the best situation, but I am sympathetic to the performance concerns of CCP to a point. Either you provide multi-region, real-time, price comparison like the US stock market or you dont - the fact that an advanced civilization cant have visibility multi-regionaly across all of EVE is well - lazy. There isnt any logical/futuristic restriction that would prevent this - have you ever checked your planetary instllations lately? (e.g its multi-regionally accessible). Dont let real-life supposed restrictions prohibit what would be an expected access to all data no matter where you are in EVE.
Bottom-line this would be very helpful - making it near-real-time is acceptable because reacting to pricing opportunities is not easy once you see them, moving products to take advantage of them takes time. Format needs to be broad - we are a multi-platform pilot community - CCP needs to decide the most easily accessible format as well as those who want minimal data, vs. those who can take advantage of maximal data.
|
Avat Ore
|
Posted - 2011.08.31 01:59:00 -
[113]
I like where this is headed.
CSV over MSSQL backup format; preferred would be xml. My personal preference is to avoid proprietary formats as much as possible.
Delay of 24 hours sounds reasonable. Let those that have their own tools continue to be rewarded for their efforts by allowing them to continue to obtain more up to date data.
Open-High-Low-Close (OHLC) would also be a great addition as mentioned by Alain Kinsella. |
Mara Rinn
|
Posted - 2011.08.31 02:16:00 -
[114]
Originally by: Hel O'Ween Please, please do not release the data in CSV format. CSV has no strong data types nor is it locale aware. It's an ugly hack, the lowest common denominator ... from a couple of decades ago.
What do locales have to do with market data exported from EVE Online? The only concern is recognising that all timestamps are in EVE time (i.e.: GMT), and that the text is e.g.: Latin 1 or Unicode. That is to say, all the information in the file comes from the same "locale" and thus the details can be agreed outside the file itself. Unicode character encoding, EVE-time timestamps.
CSV is the lowest common denominator, you have that right. It's a dead-simple format which has its problems, but none of its shortcomings will impact on the utility of the information in the proposed market summary exports.
XML adds needless bulk to the text file. MSSQL backup file is inaccessible to those who aren't running MS SQL Server.
In this instance, CSV is the simplest tool that will get the job done. "Locale" support in XML and other data description languages add needless complexity to the data export, as does packaging the data in opaque formats such as MSSQL backups.
I work with the "official" static data export, but one that was exported from MSSQL into SQLite SQL, which I then processed to ANSI-compliant SQL and imported into PostgreSQL. If that data had been available as plain CSV I would have been a much happier programmer.
[ Australian players join channel ANZAC ] |
Mara Rinn
|
Posted - 2011.08.31 02:20:00 -
[115]
For the purpose of budgetary pricing and trend analysis, three-day-old data is perfectly fine. In fact, a weekly dump produced at the same time each week would be excellent. Say during the Tuesday downtime, immediately after any patches are applied.
Anything younger than three days is treading dangerously close to "live" data. There needs to be some room in the game for players engaged in the market to carve an advantage out through their own effort.
The data of "active pilots" and "kills in this system" is up to an hour old, leaving plenty of scope for PvPers to find hot areas but still have to scout for kills. The same should apply to this market data, with the realisation that market PvP happens in a longer timescale than flying-in-space PvP.
[ Australian players join channel ANZAC ] |
Heritor Skoliya
Caldari Skoliyan Industry
|
Posted - 2011.08.31 02:29:00 -
[116]
Probably, best way is API-function that returns historic market data. IMHO. --------------------------------- SKOLI.ru - Russian EVE Online fansite |
Suki Okiwana
|
Posted - 2011.08.31 02:57:00 -
[117]
CSV please, it is universal and easy to convert. As for frequency, a daily update seems most reasonable; leaves ample wiggle room for dedicated traders and good enough for casuals.
|
Thoraemond
Minmatar Far Ranger
|
Posted - 2011.08.31 03:19:00 -
[118]
On the basis of: "Ooo! That'd be neat." I'd like current information, if just to see what would happen in the third-party space.
However, on a creating-more-niches game-design basis, I'd prefer that "current" information about the game universe require connection via the game client... so I'd vote for a delay of 48 h and the elimination of cache-scraping capabilities.
|
Rasz Lin
Caldari Uitraan Diversified Holdings Incorporated
|
Posted - 2011.08.31 04:57:00 -
[119]
Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG [ As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.
Originally by: Dr Mercy For me, there is no almost use in the data as it doesn't delineate between market activity from buy orders and sell orders.
Would you be able to provide the same data but give have a pair values for preceded by BUY and SELL to indicate buy or sell orders?
Good point.
|
TheSmokingHertog
|
Posted - 2011.08.31 05:07:00 -
[120]
Originally by: Meissa Anunthiel Edited by: Meissa Anunthiel on 30/08/2011 15:45:46
Originally by: CCP Stillman So this leaves an interesting question I'd be curious to hear. Say that we developed an API for this, which would give you the data for any set of regionID and typeID for 7 days ago. How useful would that be?
.. txt .. because the average price in a region is less important to most than the average price in a system, specifically trade hubs.
Agree with the trade hub info, but the region information is very welcome for inter-regional traders. Are those not to common in your view?
|
|
Kossaw
H A V O C Cascade Imminent
|
Posted - 2011.08.31 05:20:00 -
[121]
Edited by: Kossaw on 31/08/2011 05:22:24 CSV please. Most web based applications use MySQL (or anything BUT MSSQL) converting MSSQL to CSV is a pain in the butt for most of us.
Several 3rd party sites are already pulling market data real time from the client cache and we can get up to date market pricing within an hour or so.
So really, you need to do the whole job properly and give us an API call with updates approximately every 60 minutes or so. The data doesn't have to be instant, just reasonably fresh.
Edit: Forgot to say... Awesome. Thanks !
|
malaire
|
Posted - 2011.08.31 06:54:00 -
[122]
Originally by: Cearain But anyway, I'm not even sure how to read this. What are the item and system id numbers?
For ItemIDs, download http://zofu.no-ip.de/inca10/inca10-mysql5-sql-v1/inca10-invTypes-mysql5-v1.sql.bz2 and for RegionIDs: http://zofu.no-ip.de/inca10/inca10-mysql5-sql-v1/inca10-mapRegions-mysql5-v1.sql.bz2
Both are in SQL format, which is quite simple to parse if you just look at the datalines and ignore extra code at beginning/end.
Original thread about these datafiles: Static Data Export and Image export for Incarna 1.0
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 07:10:00 -
[123]
Slightly tangential, could someone explain to me the use of an "opening" value for this dump? (Genuine question :-))
Some people ask for OHLC values. In a 24h market like EVE's, C is not useful as it's the O of the next day. We already have H and L in this dump, leaves O. If I understand correctly, O would be the value of the first trade after 00:00 - as this is regardless of volume, what use is this value exactly?
|
Vaerah Vahrokha
Minmatar Vahrokh Consulting
|
Posted - 2011.08.31 07:39:00 -
[124]
Edited by: Vaerah Vahrokha on 31/08/2011 07:41:44
Originally by: Mara Rinn For the purpose of budgetary pricing and trend analysis, three-day-old data is perfectly fine. In fact, a weekly dump produced at the same time each week would be excellent. Say during the Tuesday downtime, immediately after any patches are applied.
Anything younger than three days is treading dangerously close to "live" data. There needs to be some room in the game for players engaged in the market to carve an advantage out through their own effort.
While for industrialists and general "buy and hold" people this is sufficient, it's still half a swing too late data for traders who need to evaluate market sentiment in a timely manner.
Basically in EvE you get:
- 0.01 ISK station traders (similar to RL "scalpers"). - 1-7 day "traditional" traders (similar to RL swing traders). - > 7 day buy and hold buyers (similar to RL stock holders).
The first type does not really need an hystorical data feed. The second type needs it and needs it in OHLC format. The third type does not need it a lot but if it was OHLC data they'd get much more information than just some averages.
With the available tools, any end user can self-create the data feature presented in this dev blog by spending 1 hour installing 2 tools and that data will be fresh and updated to the last day or less.
Originally by: Arkady Sadik Slightly tangential, could someone explain to me the use of an "opening" value for this dump? (Genuine question :-))
Some people ask for OHLC values. In a 24h market like EVE's, C is not useful as it's the O of the next day. We already have H and L in this dump, leaves O. If I understand correctly, O would be the value of the first trade after 00:00 - as this is regardless of volume, what use is this value exactly?
HLC values are the "entry point", OHLC values would be very easily produced by CCP by copying (in the export) the last day close into this day open and would save 3rd party users the hassle of importing CSV data, adding the missing datum, re-exporting it and finally feed it into the graphing applications (that generally demand full OHLC data).
Auditing | Research | 3rd Party | Collateral Holding | EvE RL Charity |
Daedalus II
Helios Research
|
Posted - 2011.08.31 07:45:00 -
[125]
I don't know if this has been asked already but there is one thing I'd like to see very much: A differentiation between sell and buy price!
The price we see now is some hodgepodge of both, isn't it? As we all know there is a difference between the sell and buy price, and that is quite important if you're a trader. It annoys me that this isn't available in the ingame market either.
___________ Interested in incursions? Join Helios Research! |
Vaerah Vahrokha
Minmatar Vahrokh Consulting
|
Posted - 2011.08.31 07:50:00 -
[126]
Edited by: Vaerah Vahrokha on 31/08/2011 07:53:10 Examples of how my own freely provided OHLC EvE markets The Forge Region data exports look like in RL finance graphing applications:
It's possible to see how using OHLC data unlocks valuable features like: candle stick bars, very easy support / resistance spotting, ability to use free time frames (daily candle sticks, weekly candle sticks and so on).
Furthermore, with OHLC you can finally use semi-professional or professional financial tools for much improved visualization. Those who wish may draw classic or proprietary indicators on EvE data and so on.
With OHLC basically a whole new universe of trading is open to you.
I have implemented a decent implementation of OHLC as a puny end user, please CCP do better than me.
Auditing | Research | 3rd Party | Collateral Holding | EvE RL Charity |
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 07:51:00 -
[127]
Originally by: Daedalus II I don't know if this has been asked already but there is one thing I'd like to see very much: A differentiation between sell and buy price!
This has been asked for a few times, but it does not make sense.
This data is market history data, not market order data. The market history stores transfers that actually happen. The way the EVE market works, for any transfer to happen, there has to be a buy order and a sell order at the same time. If you "buy from a sell order", the mechanic is that you create a buy order for that item at that price and the market fulfills that. Likewise selling to a buy order. (The only difference being no taxes for buying from a sell order right away.)
There is no difference between buy and sell orders in this data. It doesn't exist. Market history, not market orders. (Market window, "history" tab, not "orders" tab.)
|
Daedalus II
Helios Research
|
Posted - 2011.08.31 08:03:00 -
[128]
Originally by: Arkady Sadik (The only difference being no taxes for buying from a sell order right away.)
I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.
It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.
___________ Interested in incursions? Join Helios Research! |
Vaerah Vahrokha
Minmatar Vahrokh Consulting
|
Posted - 2011.08.31 08:14:00 -
[129]
Edited by: Vaerah Vahrokha on 31/08/2011 08:14:12
Originally by: Daedalus II
Originally by: Arkady Sadik (The only difference being no taxes for buying from a sell order right away.)
I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.
It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.
This is a market data dump, not a trading book data dump. I.e. you are thinking "EvE Central" while this is not that. Market happen when demand meets offer generating a trade. At that point of contact, the buy and ask became one and it's meaningless to keep track of both in a *market* data dump.
Auditing | Research | 3rd Party | Collateral Holding | EvE RL Charity |
Daedalus II
Helios Research
|
Posted - 2011.08.31 08:21:00 -
[130]
Originally by: Vaerah Vahrokha Edited by: Vaerah Vahrokha on 31/08/2011 08:14:12
Originally by: Daedalus II
Originally by: Arkady Sadik (The only difference being no taxes for buying from a sell order right away.)
I understand what you're meaning, but as you say there is also difference, at least as taxes are concerned.
It has to be possible to see if a transaction originated from a sell order or a buy order? The tax system obviously knows the difference. Then you simply keep two indices, one for orders bought from a buy order and one for orders bought from a sell order.
This is a market data dump, not a trading book data dump. I.e. you are thinking "EvE Central" while this is not that. Market happen when demand meets offer generating a trade. At that point of contact, the buy and ask became one and it's meaningless to keep track of both in a *market* data dump.
Right I got you.
So why don't we want a trading book data dump instead? It seems more useful? And if you just HAVE to have your market data dump instead, just slap the buy and sell indices together and you get that.
___________ Interested in incursions? Join Helios Research! |
|
malaire
|
Posted - 2011.08.31 08:22:00 -
[131]
Originally by: Alain Kinsella @ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).
Now that would be really interesting. No more external applications showing which orders I need to update and to what price. (At least not as easily as before, since I would need to Export each item.)
|
Kris Hakomairos
|
Posted - 2011.08.31 08:33:00 -
[132]
Originally by: Jareck Hunter About the Delay. I think i'm not only speaking for myself here, but most of the time, i'm not interrested in the price of tritanium 3 months or 2 years ago, if i want to feed my sheets for production and other things, i want to know how much something cost today, maybe yesterday or last week, so i can work with those numbers.
I think sites like eve-metrics or eve-central, got used a lot cause they provided actuall data, that could be read out automaticly.
So an update every downtime for items that get traded a lot, every 2-3 days things that get traded sometimes and maybe every week for low transaction items could be a way to go.
If you could archive something like that, that would be nice.
Perhaps a little naive.
Historical information allows you to assess long term recurring trends. For instance, if you know that historically a particular item has higher demand at a particular time of the year, you can ramp up production of that item before the demand spikes, frequently taking advantage of lower mineral prices, which means more profit for you. The in-game 1 year graph just isn't enough to determine whether a spike is a blip or an annual recurring trend - this historical information is extremely useful to savvy marketeers.
|
Gaetring Xana
Amarr Unstable Reaction Inc.
|
Posted - 2011.08.31 08:47:00 -
[133]
I'd say update it quarterly with ALL regions and systems. Perhaps more recent when some sort of API implimentation emerges.
Also, what about putting the data dump in a spreadsheet format instead of database? I know the two are similar but still..
|
Florestan Bronstein
24th Imperial Crusade
|
Posted - 2011.08.31 08:51:00 -
[134]
me gusta
|
|
CCP Stillman
|
Posted - 2011.08.31 09:11:00 -
[135]
Originally by: Rasz Lin Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG [ As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.
Services like EVE-Central are wonderful tools. But you have to understand that to generate this data, you have to correlate every single transaction in the game for any given day and any given time. They're based off orders, as opposed to the data here which is generated from transactions. There's a one to many relationship between orders and transactions. Because of that, and EVE-Central most likely doesn't have a 1hz frequency on updates from all markets and all items, the data we have to work with is many times larger.
I'd personally love to provide this sort of data live because it satisfies my interest in the data. But it's not feasible because of the severe performance impact it would have.
|
|
Erik CoolBreeze
Amarr
|
Posted - 2011.08.31 09:15:00 -
[136]
make it an api call and give us hourly updates, else why bother with this? just go to eve-central instead.
|
|
CCP Stillman
|
Posted - 2011.08.31 09:15:00 -
[137]
Originally by: Arkady Sadik
Originally by: Daedalus II I don't know if this has been asked already but there is one thing I'd like to see very much: A differentiation between sell and buy price!
This has been asked for a few times, but it does not make sense.
This data is market history data, not market order data. The market history stores transfers that actually happen. The way the EVE market works, for any transfer to happen, there has to be a buy order and a sell order at the same time. If you "buy from a sell order", the mechanic is that you create a buy order for that item at that price and the market fulfills that. Likewise selling to a buy order. (The only difference being no taxes for buying from a sell order right away.)
There is no difference between buy and sell orders in this data. It doesn't exist. Market history, not market orders. (Market window, "history" tab, not "orders" tab.)
This is the VERY IMPORTANT distinction that differentiates this data with what you might get on EVE-Central. Both are valuable, but they're vastly different in terms of quantity and thus the cost of working with the data on a computational basis.
|
|
malaire
|
Posted - 2011.08.31 09:22:00 -
[138]
Edited by: malaire on 31/08/2011 09:34:52
Originally by: Erik CoolBreeze make it an api call and give us hourly updates, else why bother with this? just go to eve-central instead.
If CCP where to provide such data, they would need to include all orders.
Eve-central doesn't do that. They most likely have one IGB-javascript-bot per region to fetch data into cache. That can fetch perhaps 30 items per minute, taking hours to fetch all items. So for any single item they only get one snapshot every few hours, and they miss all orders which are created and fulfilled between these snapshots.
edit: it seems they rely on their users botting for themselves, but the main point still stands, they will miss any orders not included in user-submitted market-order-snapshots.
|
Meissa Anunthiel
|
Posted - 2011.08.31 09:30:00 -
[139]
Originally by: Marcel Devereux
Originally by: Meissa Anunthiel
Originally by: CCP Dr.EyjoG
Originally by: Amsterdam Conversations Hey I couldn't be arsed to do the QEN, so here is your data, do it yourself.
Ahhh, you saw right through me, didn't you
The issue of the time lag of the information is a very interesting one for us. From the discussion so far I gather that anything else than 7 days is a nice to have, anything from 24 hours hold to 7 days is interesting to have, and that less than 24 hour old would be awesome.
As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Does anyone see a problem with providing 24 - 48 hour old data?
Makes it more difficult for some traders to profit by making it clearer where the opportunities are (thereby creating competition where there was little or none). Information is power when it comes to markets Eyjo, so the shorter the span, the bigger the damage.
Then again, it may also creates opportunities for enterprising individuals. For markets that have big transaction volume, 24 hours wouldn't be a big deal, for "more remote" regions, it would. Personally I think 3 days is more than enough, it gives enough information that people would have to move their arses and see what it's like "down there" right now.
And by bigger the damage you mean more tears. This is EVE. They will need to HTFU or do something else. Universal access to data and competition is good.
Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
|
CCP Stillman
|
Posted - 2011.08.31 09:37:00 -
[140]
Originally by: Meissa Anunthiel
Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.
Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.
So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?
|
|
|
malaire
|
Posted - 2011.08.31 09:51:00 -
[141]
Originally by: Meissa Anunthiel Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.
What about giving same data in-game?
For current region: all orders and full market history (as it is currently) For all other regions: delayed market history
Someone could get confused about different market histories (full vs. delayed) but then EVE is full of things which confuse someone.
|
Daid Thiefsant
|
Posted - 2011.08.31 10:17:00 -
[142]
While nice, I really miss the buy/sell price difference in this.
I live in nulsec, so I'm not a trader. Because we live deep in nulsec we need to fly in some stuff we need. Having reasonable accurate prices on those items is very useful to know, because then we'll know how much we'll spend before we fly to hisec. We can also see which loot is valuable and which is junk.
|
Aineko Macx
|
Posted - 2011.08.31 10:19:00 -
[143]
I understand this is supposed to be data on the transactions only, however, some limited/aggregated data on orders and spread would be highly welcome. For this, I suggest the following indices: - Average over the lowest sell orders equivalent to (for instance) 25-50% of the daily traded volume* - Average over the highest buy orders equivalent to 25-50% of the daily traded volume*
* this could be the volume averaged over the last week ________________________ CCP: Where fixing bugs is a luxury, not an obligation. |
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.08.31 10:20:00 -
[144]
Edited by: Dierdra Vaal on 31/08/2011 10:32:00
Originally by: Marcel Devereux This is EVE. They will need to HTFU or do something else. Universal access to data and competition is good.
I have to agree with this. The more information is available, the more interesting things people will come up with. We can already get reasonably up-to-date info through sites like eve-central so lets not pretend this sort of thing doesn't exist already. Having it done by CCP just means it's more reliable (both in data quality and security).
So I'm very much in favour of a 24h (best) or 48h (acceptable) delay. Die hard traders can still get to the good deals first with this delay while 3rd party developers can still get reasonably accurate prices.
Originally by: CCP Stillman So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?
I rather doubt it tbh. Judging by some of the posts in this thread and the availability of services like eve-central, the super serious traders already have all sorts of financial graphs at their disposal. This might make their data a little more reliable but it doesn't seem to give them anything they don't already have.
And yet, even with these trading behemoths, plenty of small time traders (who aren't quite as srs bsns) manage to flourish just fine.
On top of that, we've seen that quite quickly market analysis tools are made public. I'm sure some people have their own spreadsheets and tools, but there are enough developers in the community who will release their own tools (out of altruism, ego or in the hopes of getting ISK donations, etc). As such I don't think there is a real risk that releasing info at a 24h interval means a few super srs bsns traders will walk all over the rest.
And the benefit is obviously all sorts of interesting things the eve community will no doubt come up with, that might not be possible (or not relevant) if the delay was longer.
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.31 10:27:00 -
[145]
Originally by: Rasz Lin Edited by: Rasz Lin on 31/08/2011 05:05:18
Originally by: CCP Dr.EyjoG [ As CCP_Stillman mentioned then there is a real technical challenge to have close to live data. That challenge will not be resolved soon so let's focus on the 24 hour+ option.
Perhaps CCP should consider hiring someone from eve-central or eve-metrics, as they have/had NO PROBLEM generating this data live 24/7 with no delay.
I can tell you that this was only possible because of the low volume of uploads they received. I now do the same thing with EVE Marketeer, and it takes about 0.2 seconds to update the historical data for an item, so as long as I'm at less than 5 updates a second I'm ok (could be faster, it isn't optimized at all). Imagine doing this for the entirely of the EVE universe every time someone updates an order somewhere and it should give you an idea of the scope.
Originally by: CCP Stillman So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?
Not necessarily since a lot of us provide these tools for free. In fact, I think the best tools that are in use are free (but then again, I cannot see what tools are not available to the general public). The problem is mostly making people aware that they exist ---
|
hero honda
|
Posted - 2011.08.31 10:37:00 -
[146]
can we get an open and close with that? (start of trading day = end of down time,end of trading day = beginning of down time)
|
hero honda
|
Posted - 2011.08.31 10:40:00 -
[147]
BTW csv, I can turn csv into whatever format I want easy and parsing a csv is easy.
|
Alain Kinsella
Minmatar
|
Posted - 2011.08.31 10:59:00 -
[148]
Originally by: malaire
Originally by: Alain Kinsella @ Stillman - Not to put a wet blanket over the good news, but is this preceding any news of banning cache readers as bot activity? Skreegs has not yet published his expected devblog on what you will consider botting, and supplying market history data seems to be a step towards forcing market data to 'official' channels only (market dump button, and now this).
Now that would be really interesting. No more external applications showing which orders I need to update and to what price. (At least not as easily as before, since I would need to Export each item.)
I've seen it before (with SL and introduction of their LindeX(tm) exchange - my own little 'I was there' moment actually ).
As discussed here though, I think what we're agreeing on currently will not hurt the loss of cache reading (unlike what happened with LL). Would still like an answer though.
|
Meissa Anunthiel
|
Posted - 2011.08.31 11:11:00 -
[149]
Originally by: CCP Stillman
Originally by: Meissa Anunthiel
Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.
Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.
So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?
Available of "mass" data is always very interesting for those who have the means to data mine it, so there's a lot of arguments in favour of these exports (which I support fully).
Smaller/casual traders (which I think are more numerous than the "pros") can benefit in areas that are more "niche" and thus require less micromanaging such as (but not limited to): - small volume items in trade hubs or missioning centers - empire region trading (buying in Jita, selling in Amarr) - risky business (lowsec/nullsec trading)
I know a few people who make quite good money buying in Rens and reselling in Hek or Teonusude for instance (6 jumps from Rens, but in another region).
These markets are discovered through investigation by moving around and checking prices of individual item, something dumps do away with. A long trail of data gives "investigation clues" so data miners still need to verify if the markets are still valid by actually moving there. Short trail provides that data without moving about by showing if the decrease in average/median price is very recent.
Someone with an automated data mining system can easily identify new markets if too many people come into his juicy one, but the trader without those tools will require a significant amount of manual investigation to find something, and by the time he does, that market may already be gone.
My argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail (Look what sub 100ms data feeds have done to the stock market as an extreme of what I'm saying). And while I favour giving the people "clues" to investigate, I'm not sure about what full automation would do.
I'm sure Eyjo would love to have sub 100ms market feeds and full automation capability as a research project (I'd love it too, personally), but I'm thinking of the noobs here. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Di55y
|
Posted - 2011.08.31 11:21:00 -
[150]
I think you, CCP, should think the other way round:
Deliver as fast as possible as extensive as possible. People will find a reasonable meaning for this data (1+ week/month old data will find its way into blogs, etc. newer data will make profit for nifty traders).
That said, you can start thinking about the how: - API ? - Balancing via "what items, regions, stations are interesting for most of the players" -> kind of hit counting / whats popular
So you essentially will have to make your hamsters run at maximum (ridiculous) speed and delay (exactness) will vary for different markets.
Result is: - one will have almost instant access to tritanium (or ore) prices in jita - oen will have to rely on jumpfreighter prices which might be a week old (or look it up ingame, via eve-central or co.)
After that, you can think about giving your hamsters +5 nano-leg implants.
|
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 11:28:00 -
[151]
Originally by: Meissa Anunthiel My argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail
Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...
The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.
You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).
Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.
(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)
|
El Pepito
|
Posted - 2011.08.31 11:32:00 -
[152]
I would like to make it possible to download only part of the data : we have to choose the item and the range of the data to download. ex : tritanium, from 1 may 2010 to 26 june 2011. like this, it would be possible to analyse more simply in spreadsheets. Here, the files is just too big... My version of open office open just the first million of line in .CSV and nothing in .SQL!
But I say Great
|
Meissa Anunthiel
|
Posted - 2011.08.31 12:07:00 -
[153]
Edited by: Meissa Anunthiel on 31/08/2011 12:08:37
Originally by: Arkady Sadik
Originally by: Meissa Anunthiel My argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail
Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...
The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.
You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).
Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.
(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)
It takes you 4 hours to dump the data for a region, you have tools to parse the cache, the more casual traders don't. I know you already played with manufacturing and market many years ago, remember how you did things then, and realize that the less devoted and/or newer players do it like that now. That play style needs to be preserved as well, and if something can foster the other ones without needlessly harming the casual players, then absolutely.
I'm arguing that the advantage provided by this data will be available to a certain group (they're not few, they're just the dedicated ones), not that it should. Eve Central is regularly unreliable, they provide indications that need to be investigated, just like I argue this dump should.
Strictly speaking, my position would require that no dump exist, but the added value is too great to pass up on, now I'm arguing for some mitigation. We can start with 3 days easily, see what comes out of that, see the impact it has on the smaller traders, and based on that lower it further. It's easier to go from 7 days to 3 to 1 to sub-1 than it is to revert from sub-1 to 7. My point is caution...
As an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).
Edit: And if you wonder why I spend time arguing against my own preference, it's because it's my job as member of the CSM... I really understand and agree with your arguments. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Eybium
Hephaestus Shipbuilding
|
Posted - 2011.08.31 12:26:00 -
[154]
Originally by: CCP Stillman I'd personally love to provide this sort of data live because it satisfies my interest in the data. But it's not feasible because of the severe performance impact it would have.
In the real-world, I'm actually a high-level technical expert for a major data consulting firm, so I have experience drawing reporting and migration dataloads off of high-activity-volume multi-terabyte systems. Depending on the backend architecture you're using for your live data, there may be a way to get the near-live (or at least sub-60-seconds) feed you're looking for. Whether or not you're looking to publish a feed at that resolution, I imagine it might help out CCP's internal analysis by yourself and CCP Dr.EyjoG.
If you're interested in having a feasibility conversation on using some of the latest techniques available to get that kind of information in that kind of time, send me a message on EVEGate and we'll take that discussion offline.
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 12:33:00 -
[155]
Originally by: Meissa Anunthiel As an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).
I am not arguing for myself.
I already have this data. I do not need it.
I find it bad game design that the ability to write webpages that browse the market and programs that read cache files (my thanks to Entity here) and upload them to server gives me a tangible in-game advantage.
Hence, I would like to mitigate the advantage I can get from being able to use all this arcane stuff and have CCP provide the data I can get anyhow. So that it's not just the geeks with no life get to actually benefit from this data.
1-day old data is great. 2-day old data, ok, too. Older than that and the values become slowly useless for our use and we'll just keep using our existing toolchain - meaning we "dedicated users" keep having an advantage over the "less dedicated" user, instead of losing that advantage.
Quote: I know you already played with manufacturing and market many years ago, remember how you did things then
We used to use the QUANT Corp 7-day index (QTC-7), which was generated by some poor sod flying around the three Minmatar regions every day and typing down market values by hand. I think he later switched to the "market export" button.
When that poor sod burned out from that work, we switched to EVE Central, who were (and are) uploading order data from the "export" in-game button. Later still we switched to EVE Metrics until they shut down, and then back to EVE Central. As EVE Central is quite unreliable and does not offer market history data, only order data, we looked for a replacement. eve-marketdata.com looked good, but we got tired of being dependent on a 3rd party site, so we implemented our own cache reader and uploader instead.
It got easier over time, but not that different.
|
Taedrin
Gallente Kushan Industrial
|
Posted - 2011.08.31 12:36:00 -
[156]
Originally by: CCP Stillman
Originally by: Meissa Anunthiel
Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.
Being a programmer and market dude myself, I can think of numerous ways to gain an edge off this data. But then again, as pointed out, there's already ways of scraping very similar data in-game, though less efficiently.
So I'd be very interested in hearing any arguments against provide data every, say 24 hours. Would that ruin it for the small guys, who can't do these sorts of tools?
If you release this data, you will be ruining someone's (EVE) life somewhere and sometime. This is REGARDLESS of the time delay involved,
That being said, the more timely this data is, the more useful it is. Knowledge is power, and this sort of data will help grease the wheels of economy. Yes, it may put the squeeze on profits, but it may also help orders to fill faster.
Anyways, the requirement to use third-party tools is already pretty much standard from the industrial aspect, and I bet that most traders use third-party tools already anyways. I imagine that it won't be too long after you make this data available that developers will include the option of using this data in their tools. ----------
Originally by: Dr Fighter "how do you know when youve had a repro accident"
Theres modules missing and morphite in your mineral pile.
|
TheButcherPete
StoneWall Metals Productions
|
Posted - 2011.08.31 12:44:00 -
[157]
I like how the market is turning into the Dow Jones Industrial Average, good going CCP.
-Pete
|
malaire
|
Posted - 2011.08.31 13:02:00 -
[158]
I made simple perl-script to filter the data, feel free to (ab)use as you want, if you find it usefull: http://pastebin.com/uPpnNCh8
|
Aineko Macx
|
Posted - 2011.08.31 13:33:00 -
[159]
Is the averagePrice calculated correctly by weighting each transaction price according to quantity? ________________________ CCP: Where fixing bugs is a luxury, not an obligation. |
Cearain
Caldari The IMPERIUM of LaZy NATION
|
Posted - 2011.08.31 13:41:00 -
[160]
Edited by: Cearain on 31/08/2011 13:41:29
Originally by: Meissa Anunthiel Provide the exact same information in game without a person having to travel between regions and I'd agree. Requiring people to implement databases, API calls and other such things is unreasonable. Personally I have no issue with it, I'm a developper, but not everyone is, and those people matter too...
This API/export differs from other APIs in this that it provides information that isn't readily available to people in-game, that's a big difference.
Meissa I agree with your concerns. EVE is sometimes called "spreadsheets online." Its clear many people in this thread are fine with that, but I am not sure it is a great selling point for most. Its definitely not for me.
I don't want to be forced to run multiple out of game apps or analyze a bunch of out of game spread sheets in order to be competitive in eve. Thats where this is heading.
Give information in game and in english. -Cearain
Make fw pvp not pve http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1329906&page=1 |
|
Marcel Devereux
Aideron Robotics
|
Posted - 2011.08.31 13:44:00 -
[161]
Originally by: Meissa Anunthiel Edited by: Meissa Anunthiel on 31/08/2011 12:08:37
Originally by: Arkady Sadik
Originally by: Meissa Anunthiel My argument is, in essence, that longer trail still requires manual checking, shorter trail does away with that need. The barrier between people without those tools and those with it will increase the shorter the trail
Well, ignoring the fact that the market history is much less useful for finding "trade opportunities" than order data is...
The problem here is that this is not about "giving them the data" vs. "not giving them the data." The data is already available. It takes about 4h to do a full market dump for a region, and having 3 accounts to do that during your work day isn't particularly difficult. Come home, work with current data.
You are, in essence, arguing that the advantage of this data should be available only to a few. This move from CCP makes that data available to more people. Making the data readily available will also make it easier to incorporate into websites that help everyone (e.g. eve-central).
Giving out this data in a more structured form helps newer (or "less focused") players, as opposed to harm them.
(Oh, and just to make this clear, I'm talking about daily data, not sub-daily data. If you go below 1-day resolution, you start favoring people who can play at any time of the day as opposed to people who happen to have to go to work and such.)
It takes you 4 hours to dump the data for a region, you have tools to parse the cache, the more casual traders don't. I know you already played with manufacturing and market many years ago, remember how you did things then, and realize that the less devoted and/or newer players do it like that now. That play style needs to be preserved as well, and if something can foster the other ones without needlessly harming the casual players, then absolutely.
I'm arguing that the advantage provided by this data will be available to a certain group (they're not few, they're just the dedicated ones), not that it should. Eve Central is regularly unreliable, they provide indications that need to be investigated, just like I argue this dump should.
Strictly speaking, my position would require that no dump exist, but the added value is too great to pass up on, now I'm arguing for some mitigation. We can start with 3 days easily, see what comes out of that, see the impact it has on the smaller traders, and based on that lower it further. It's easier to go from 7 days to 3 to 1 to sub-1 than it is to revert from sub-1 to 7. My point is caution...
As an individual, I'd rather have instant dump of anything I request because I can develop anything needed to take advantage of it, but I'm arguing for the numerous less dedicated players (who most likely won't even post here).
Edit: And if you wonder why I spend time arguing against my own preference, it's because it's my job as member of the CSM... I really understand and agree with your arguments.
You keep saying that the smaller trader either won't be dedicated enough to mine the data or won't have the technical know how. You haven't taken into account in your counter argument the creation of 3rd party software to help the less dedicated and non-technical players. I can assure you that someone will create tools to help players mine this data. Therefore I propose we stick with the 1d resolution ;-)
|
Arablue
Minmatar BSC LEGION Split Infinity.
|
Posted - 2011.08.31 14:36:00 -
[162]
Originally by: CCP Stillman
Originally by: Dierdra Vaal
As such, is it possible that this will be upgraded to an API call? Where, similar to eve-central, I can query one or more itemIds (as well as location parameters, time parameters, etc) and get their up-to-date price statistics back?
Originally by: Arkady Sadik
Have you considered providing an API interface instead of file dumps?
It's no coincidence that the API team has been working with Dr. EyjoG and his team on this. Our ultimate goal is to expose this through an API with more up to date data. This release is to gather feedback and find out how you guys can best benefit from a such potential API.
Definitely csv until it gets worked out with the new API key going live.
|
Avall
|
Posted - 2011.08.31 15:48:00 -
[163]
On data delay: I don't believe up to date data will only benefit hardcore traders since easy to access 3rd party tools will spring up quickly. However, the point that up to date prices may overly benefit players with a lot of play time on their hands is well made. With a 24h delay these players could still gain their deserved advantage using in-game checked eve-central data correlated with the delayed CCP data, without leaving less hardcore traders too far behind. With this in mind, a 24h delay may well be the perfect compromise.
Also remember that since CCP data does not include outstanding buy and sell orders, they good-deal-between-regions trader play style is more or less preserved as is.
A 24h delay would be good enough to spot larger market trends as they happen. 24h is also perfectly suitable for item price checks and killboards. However, already at a 48h delay, the value for these uses are greatly diminished.
On data formats and delivery: A daily csv dump is fine for market data mining tools and trading desktop applications. However, for mobile applications, a 1mb load over a shaky 3G or edge connection may take a very very long time, even 100kb will introduce a large delay. And to use price data directly in interfaces, a quick API call per item (or several items) is needed. If API server spamming is an issue, cashing should help and even rate limiting calls to the API could work depending on implementation (say maximum 10 calls per whole minute with maximum 100 total items requested, this would allow for priced item listings without enabling to load prices for "all items" via the API).
Again, good work CCP!
//Avall
|
Black Romero
|
Posted - 2011.08.31 17:28:00 -
[164]
Once what was EVE Online then Spreadsheet online has now turned into Database online? :)
I will stick to small niche trading that doesn't require me to re-live my previous career in computer databases and hope this data doesn't run me out of business in all markets.
Now I am off to go blow stuff up. Peace.
|
Barakach
|
Posted - 2011.08.31 17:34:00 -
[165]
Edited by: Barakach on 31/08/2011 18:04:35 Edited by: Barakach on 31/08/2011 17:37:39 Is it possible to get this data also grouped on buy/sell? I'm sure the average buy and quite different from the average sell.
Side note: Do people not realize that you can get MS SQL Express for free? SQL is very powerful.
Edit: could we possibly get a Standard Deviation with that? I would love to be able to recreate a bell curve showing the relationship to price and volume.
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 18:08:00 -
[166]
Originally by: Barakach could we possibly get a Standard Deviation with that?
This is actually a very good point - an average without standard deviation kind of loses value. :-)
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.31 18:17:00 -
[167]
Edited by: Callean Drevus on 31/08/2011 18:17:36
Originally by: Barakach Side note: Do people not realize that you can get MS SQL Express for free? SQL is very powerful.
Do people not realize that MS SQL Express only runs on windows?
Quote: Once what was EVE Online then Spreadsheet online has now turned into Database online? :)
It was from the beginning :P this is what attracted me about EVE. ---
|
Isabella Thresher
Fat Kitty Inc.
|
Posted - 2011.08.31 18:22:00 -
[168]
i prefer the SQL version, sql management studio is an awesome tool, and easy to use. start sqlmgm studio, restore from backup file, done thanks for this!
|
Hel O'Ween
Men On A Mission EVE Trade Consortium
|
Posted - 2011.08.31 18:30:00 -
[169]
Originally by: Arkady Sadik
Parsing CSV (especially as this dump primarily contains numbers only, no complex strings) is trivial and fault-free in comparison.
Originally by: CCP Stillman
On the format discussion, I want to propose a suggestion. It might be silly, but hear me out:
CSV is a very simple format. I know that at least the database engine I work with on a daily basis(MSSQL), can import a CSV file into a table in about a small line of SQL. And I'd be surprised if other database engines couldn't.
Until you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.
As I said: for a text-based format almost anything is better than CSV. -- EVEWalletAware - an offline wallet manager |
Lolmer
Amarr Merciless Reckoning Executive Solution
|
Posted - 2011.08.31 18:31:00 -
[170]
Edited by: Lolmer on 31/08/2011 18:35:20 Weekly (or monthly, but prefer daily for granularity (but not too granular)) data dumps and use CSV.
More people will be able to use CSV (MS Excel and Google Spreadsheet both know how to use it) whereas the SQL dump many people won't know. Now, the people who could use the SQL dump very well will also be knowledgeable enough to import the CSV into their own SQL (or otherwise) DB.
- CSV serves everyone.
- SQL dump serves a limited group.
|
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.31 18:47:00 -
[171]
Originally by: Hel O'Ween Until you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.
Not exactly true. This is only an annoyance when you have to work with a lot of undocumented CSV files in different formats. If you know what you can expect it isn't very difficult to make sense of it. Nor an annoyance to work with. And I just do not imagine CCP will change the format every time they make a dump ---
|
Arkady Sadik
Minmatar Electus Matari
|
Posted - 2011.08.31 18:55:00 -
[172]
Originally by: Hel O'Ween As I said: for a text-based format almost anything is better than CSV.
I'd say SQL is actually worse, considering the differences between the various SQL dialects. Converting a nice CAST(0x96040474 AS SmallDateTime) to a DATETIME value that other SQL servers can understand takes quite some gymnastics (that's from an MS SQL export). And it's just a very obvious extreme example.
Telling your csv library to use "," as a column separator (usually the default) and "\n" as a row separator (usually the default), with no escpaing necessary, is trivial in comparison, really.
(Again, as soon as you have commas or newlines in your records, you really want something else than csv, but this is not the case here. So pretty much any halfway sensible format works with very little overhead, compared to the .BAK file.)
|
Monstress
|
Posted - 2011.08.31 19:51:00 -
[173]
Originally by: Hel O'Ween
Originally by: Arkady Sadik
Parsing CSV (especially as this dump primarily contains numbers only, no complex strings) is trivial and fault-free in comparison.
Originally by: CCP Stillman
On the format discussion, I want to propose a suggestion. It might be silly, but hear me out:
CSV is a very simple format. I know that at least the database engine I work with on a daily basis(MSSQL), can import a CSV file into a table in about a small line of SQL. And I'd be surprised if other database engines couldn't.
Until you happen to use a locale where "CSV" is quite different. I work for an international company and have the "pleasure" to deal with exchanged CSV files. You know, in Germany in a "CSV" file, columns are actually delimited by ";" due to the fact that the comma is our decimal separator. Now try making sense out of a "real" CSV file.
As I said: for a text-based format almost anything is better than CSV.
You do know that databases that do support importing CSV files also support specifying custom delimiters, line endings, quotation and escape characters right?
|
FuzzyLogik360
Caldari
|
Posted - 2011.08.31 21:07:00 -
[174]
I suspect this has already been captured by someone else already, but it is worth reiterating. Please can we have Avg./Median price differentiated by BUY and SELL orders seperately. Thanks.
Oh, and another thing, I think the optimum delay is something around 5 working days. Why? Well we don't want the market to be too efficient, as otherwise, as any good economist will tell you, prices will (in theory) collapse to the base cost value. If that happens, then no-one will be making any money. I suspect for 95% of items, prices won't move very much in 5 days anyway. For the remaining 5%, that's what EVE-Central is for.
|
K Kerryngktonn
|
Posted - 2011.08.31 21:39:00 -
[175]
I suspect buy/sell cannot be differentiated here because of the internal implementation (the transaction is always two orders meeting, the outstanding one and the immediate one, so that's a Buy and a Sell at the same time, afaik), so we won't have such kind of details.
As for the data format: CSV for the dump as it's broadly compatible. I'm not sure about the necessity of the API calls, because if we're talking about the delayed data (a week, a month or so), then we're not really interested in the latest two days of the data, but the whole data chunk. This is not much useful for short-term planning as had been already mentioned. In my opinion, it's all about pinpointing mid/large economic cycles or something analogous to real-world economy.
On the other hand, maybe it's possible to implement these data collection routines DT-time and collect the aggregate of the finished "day"? I don't really know about the perfomance cost here, but it seems justified - there's no transactions going on, the previous financial day is closed, so why not spend two minutes on gathering the data? That way the data is really actual, and if we jack in the API here, we'll get a pretty distilled version of Awesome.
|
Xander Hunt
Minmatar Dead Rats Tell No Tales
|
Posted - 2011.09.01 02:03:00 -
[176]
*GRUDGE* So I'm really confused about this whole thing....
CCP hasn't said anything more aside from the "The $99 `is coming` for using our information to protect our IP" yet, the developers go to the players for useful information? Sounds to me like they're trying to put out more and more tools (Useful or not) to make that cost seem more worth while. I'm not paying $99 when I ask for nothing back, or, a small in-game currency donation.
Don't you guys have accountants that sit in development meetings to see how much more can be milked out of your players? Couldn't you just take them aside and say "Hey, what would you find useful in a historical price dump?" then shove it down our throats like pretty much everything else that has been dumped into this game over the past year or so COMPLETELY unannounced?
The data you get from eve-central alone, including the archived data, can be derived into what CCP is "willing" to offer, not to mention E-Cs data is more "real time" and has a hell of a lot more history, however not 100% accurate and up to date on all infrequently used items since its user supplied data, but I'm sure there are regular updates to at least the basic ores. The data from E-C can be massaged into different manners to suit what is required for the job.
The *ONLY* issue I have with E-Cs data, and its not their fault really, is that you're never told when 100% of an order is sold out. Aside from that, I cannot fathom how this additional tool from CCP is going to be any better than what is already derived out there.
That $99 thing really freak'n blows, and this kinda thing doesn't sweeten the pot any.
*DEV* Even with the above mentioned, I'm still going to give my dev advice.
As for the commentary about CSVs... International or otherwise, since you're working purely with numbers, and two delimeters, if you're space and download size conscious, you can work the data into 4-bit fields and still have room for delimiters. But, since nothing will easily read a 4-bit character file, 8-bit characters will do, its standard, and its been around since the first 8-bit computers were made. Forget uni-code, forget double-character notation, deal with CHR(0) to CHR(255), one byte per character, and you eliminate problems converting to and from chinese, russian, english, french, spanish and every other different character type notation.
Stay away from using SQL structured language as the primary dump source. Any recent DBMS will easily read in the delimited file, and drop it into a table. Hell, even regular programming languages can deal with the file, line by line, split up the data due to the delimiters, and process the data accordingly. There shouldn't be any question about this.
|
Barakach
|
Posted - 2011.09.01 02:49:00 -
[177]
"I'd say SQL is actually worse, considering the differences between the various SQL dialects. Converting a nice CAST(0x96040474 AS SmallDateTime) to a DATETIME value that other SQL servers can understand takes quite some gymnastics (that's from an MS SQL export). And it's just a very obvious extreme example."
Of course casting raw binary data into a value type that is stored differently amoung SQL implementations will give different values for different implementations. |
Barakach
|
Posted - 2011.09.01 02:52:00 -
[178]
"I suspect buy/sell cannot be differentiated here because of the internal implementation (the transaction is always two orders meeting, the outstanding one and the immediate one, so that's a Buy and a Sell at the same time, afaik), so we won't have such kind of details."
Would be sad if they can't even track info that simple about an order.
|
Somatic Neuron
|
Posted - 2011.09.01 03:28:00 -
[179]
Just a few things here....first and foremost...this would have helped me a lot more several years ago...but, it's about time ;)
Second, XML is the way to go on this. Not CSV nor SQL. Standard XML rowsets would be fine and dandy.
Third, API calls, yes, please. Previous day's generated during downtime....all others available on a per call basis (they should be cached results, so cheap and easy...) pass in a date, and optionally a region. Probably the best would be another API call, that you could pass in an itemID, date range and optionally a regionID, to obtain only the items you are interested in....just brainstorming here... ---------- |
Barakach
|
Posted - 2011.09.01 03:50:00 -
[180]
"Second, XML is the way to go on this. Not CSV nor SQL. Standard XML rowsets would be fine and dandy."
For the API, XML is fine, but for data dumps, XML is just a great way to bloat up file sizes. They're not doing CSV vs SQL, they're offering both.
It just saves me time from having to import data into SQL. |
|
Vaerah Vahrokha
Minmatar Vahrokh Consulting
|
Posted - 2011.09.01 06:36:00 -
[181]
Edited by: Vaerah Vahrokha on 01/09/2011 06:39:56 Edited by: Vaerah Vahrokha on 01/09/2011 06:38:32
Originally by: Somatic Neuron Just a few things here....first and foremost...this would have helped me a lot more several years ago...but, it's about time ;)
Second, XML is the way to go on this. Not CSV nor SQL. Standard XML rowsets would be fine and dandy.
Third, API calls, yes, please. Previous day's generated during downtime....all others available on a per call basis (they should be cached results, so cheap and easy...) pass in a date, and optionally a region. Probably the best would be another API call, that you could pass in an itemID, date range and optionally a regionID, to obtain only the items you are interested in....just brainstorming here...
The whole world back ends (those that really HAVE to work) are still running on good ole comma / tab delimited files. XML is bloated, requires more work before extracting the data and is more useful for frequently changing schemas than a dump that is the same since 8 years ago.
Also, to the guy who said MS SQL has a free edition: yes but it pumps so MUCH garbage in the computer it's awful. It's so awfully bloated and large for a task (importing EvE files) you could easily do with SQLite, that happens to be faster, smaller, portable AND really free.
As 3rd party EvE app developer, I will never ever accept to install MS SQL or to write an EvE application requiring such software to work. The guy who did so (EMMA EvE trading software) got so many belly aches you have just to read its troubled history on MD forum to learn how bad it is. I also maintained EMMA for a short while and indeed I basically had to reformat the computer that got infected with those many gigabytes of partially uninstallable garbage.
Auditing | Research | 3rd Party | Collateral Holding | EvE RL Charity |
Kane Molou
|
Posted - 2011.09.01 08:38:00 -
[182]
While I find this all nice et etc.. It gets me why CCP can't do something simular to what real life trading groups do both in and OUT of game...
In real life if i'm trading stock, shares, vast quanities of grain etc etc etc.. I go look at the current 'market' this market doesn't list off every purchase currently being done.... instead it shows me the various major market regions on our planet and the rolling average for the current trade price.. now this is typically delayed by about 15 - 30minutes.. no more..
Why is it delayed? Because it takes a little bit of time to run the data through a average(a1-a999999) to get done... But the point is that this data is then made avalible FREE to anyone in the Public Market.. be it Forex, Cattle trading, Grain Trading what ever and it's made avalible no matter WHERE in the world you are.
All this new data for eve is great yes but at the same time it's not it's majorly out dated and only helpful if you want to see the same info that a certain some one is meant to do for us anyway, you know the reason CCP employees him in part.. which is to give us historical market trends. In other words useless for the average trader, and actually even more useless really for the long term trader.. why? Well lets go and look at a few things:
first off we the players can only easily access our small regions market at any time so unless you have a bunch of alts etc or your using websites which may not even be up-2-date anyway.. your unable to do any real long term planning as a 'casual' trader, the ones which do offer you great instant access to data etc want isk or the like for it..
In this respect it's not like real life, which sucks.. I can't just go and get the data from the foreign exchange or the local bank or the local wheat board etc for say 4 regions away.. which is again stuipid heck apparently a region can't even talk to it's neighbour! Historical data is not going to help me here, because we all know that while Megathrons might have been selling for 100mil in Jita 3 months ago, they could be selling for 60mil now and have been for months anyway..
then there is the other reason this data is normally made public for historical viewing.. which is called investor relations and the like.. basically in real life I have this thing called the 'stock exchange' something eve obviously ONCE had in mind because we have the ability to make shares but then of course like everything it apparently was meant to be player run or something but we never got the tools to do it 'fairly' or the like.. and no one relly has the power to buy up enough stock to start one either.. but i'm degrading from the ponit ..
In real life, before a 'large' investor goes and buys stock in a company they would go and do their homework.. what homework is this? well normally they would look at what the company they are buying stock in produces or does.. so lets say Mining High Sec 4 Us. Now this company mines High Sec ores in *thinks* Genesis primarly Trit, Pyrite, Mex and Iso selling it to the market in this region.. I would as an investor look both at the 'market' for what they are selling, so i'd see if those minerals are going up, down or level in sales, then i would obviously look at the annual reports from the company and see if they made or lost money etc.. then based on all of this I'd go buy shares..
For something like this 3 month old data heck even day old data is FINE..
For Trading however, it is not.
As Dr EyjoG likes to point out Eve is a living Breathing Entity that tends to follow real life market trends.. it has orders being filled every minute of every day (minus down time).. so unless the data is avalible with only a 'small' delay like real life there is no point to it.
of course now i expect no answer to this forums die tonight again.
|
Meissa Anunthiel
|
Posted - 2011.09.01 08:50:00 -
[183]
Stillman, Eyjo,
After much thought on the subject, while I still think the concerns I voiced are valid, I believe there will still be plenty of venues for casual traders to thrive in, mostly in the low-volume items and dangerous locations, and the advantages of "current" data outweigh the downsides. So go on and implement it with as short a delay as possible, even 1 hour :p, as long as there's no individual orders exposed and only average/median/volume/etc. ----- Member of CSM 2, 3, 4, 5 and 6. My blog
|
Hel O'Ween
Men On A Mission EVE Trade Consortium
|
Posted - 2011.09.01 10:50:00 -
[184]
Originally by: Monstress
You do know that databases that do support importing CSV files also support specifying custom delimiters, line endings, quotation and escape characters right?
Yes, I do know. But the typical user trying to open it in a spreadsheed not.
Bah, why do I still care at all?. My sub expires in December and that's it for me. -- EVEWalletAware - an offline wallet manager |
Lukas Rox
Torchwood Archive
|
Posted - 2011.09.01 11:37:00 -
[185]
I have restored the backup and tried to link it to existing data from the usual data dump.
Unfortunately the values from column "stationID" in the Marketdata db don't match with values in the "staStations" table from Data Dump: JOIN between tables on stationID results in 0 records selected
The values from Marketdata.stationID are however similar to values from column staStations.corporationID
Is this a mistake in the backup or am I wrong here? ---
Faction Ships | Sansha NPC |
Pyus
Hand Of Midas
|
Posted - 2011.09.01 11:45:00 -
[186]
Originally by: Lukas Rox I have restored the backup and tried to link it to existing data from the usual data dump.
Unfortunately the values from column "stationID" in the Marketdata db don't match with values in the "staStations" table from Data Dump: JOIN between tables on stationID results in 0 records selected
The values from Marketdata.stationID are however similar to values from column staStations.corporationID
Is this a mistake in the backup or am I wrong here?
It's actually a region ID so do a join on tha and you should be good. All the data is per region info just like the in-game History tab when you're viewing an item.
|
Lukas Rox
Torchwood Archive
|
Posted - 2011.09.01 11:49:00 -
[187]
Originally by: Pyus
Originally by: Lukas Rox I have restored the backup and tried to link it to existing data from the usual data dump.
Unfortunately the values from column "stationID" in the Marketdata db don't match with values in the "staStations" table from Data Dump: JOIN between tables on stationID results in 0 records selected
The values from Marketdata.stationID are however similar to values from column staStations.corporationID
Is this a mistake in the backup or am I wrong here?
It's actually a region ID so do a join on tha and you should be good. All the data is per region info just like the in-game History tab when you're viewing an item.
yup, that works just fine. Column names are misleading :P ---
Faction Ships | Sansha NPC |
|
CCP Stillman
|
Posted - 2011.09.01 11:54:00 -
[188]
Originally by: Lukas Rox
Originally by: Pyus
Originally by: Lukas Rox I have restored the backup and tried to link it to existing data from the usual data dump.
Unfortunately the values from column "stationID" in the Marketdata db don't match with values in the "staStations" table from Data Dump: JOIN between tables on stationID results in 0 records selected
The values from Marketdata.stationID are however similar to values from column staStations.corporationID
Is this a mistake in the backup or am I wrong here?
It's actually a region ID so do a join on tha and you should be good. All the data is per region info just like the in-game History tab when you're viewing an item.
yup, that works just fine. Column names are misleading :P
Yep, I messed up. Blame me :D
|
|
Haguu
Caldari TLA Ltd
|
Posted - 2011.09.01 14:15:00 -
[189]
This is very exciting news and TYVM.
Originally by: Meissa Anunthiel Stillman, Eyjo,
After much thought on the subject, while I still think the concerns I voiced are valid, I believe there will still be plenty of venues for casual traders to thrive in, mostly in the low-volume items and dangerous locations, and the advantages of "current" data outweigh the downsides. So go on and implement it with as short a delay as possible, even 1 hour :p, as long as there's no individual orders exposed and only average/median/volume/etc.
I am not as clear on the problems this would cause the casual traders. The professional traders, organizations, and b*ts currently have this information, and then some, via the cache. I thought the objection is that these players would prefer to continue to have an advantage over the masses? Once you read the cache, then you have complete control over it - upload to a website, send a TCP/mail message to another program/client to process, etc. My understanding is that you could have your child/secretary have their own account, they periodically create a cache your software reads and then contacts you IG or OOG that your attention is required. I could know I don't need to log in to babysit my orders because if there were an issue, the last time my secretary updated the cache my software would analyses the data, sense there is an exception condition, and contact me. Thankfully, this is EVE and nobody would ever violate the TOS and automate something based upon these cache values. Am I overlooking something? As soon as cache reading was allowed, it seems like the options the pros have now dwarf giving the casual player some sales data a few hour old?
-----
I know it is different data, but I really like eve-central and I think a lot of people rely on it, especially after the EM shutdown. If I were a CCP employee, it would worry me a bit that something a lot of my customers use is provided by one single individual that no longer plays the game. I.e., I think more data coming from CCP is a Good Thing.
-----
My opinion is that XML is probably the "proper" answer and CSV would be smaller, faster and more convenient. Obviously, JSON is absolutely forbidden since that is what Blizzard used for their market data downloads. But again, no amount of free MSSQL tools are going to help those of use not using windows or those against proprietary, and probably version-dependent, formats.
|
Zedia Zhane
|
Posted - 2011.09.01 15:31:00 -
[190]
There are (or have been) several sites that give access to market data on a 24-hour delay. These sites provide APIs to allow people to download the data into their own applications for analysis.
I'm talking about Eve Central, Eve HQ, Eve Marketdata, and the now-defunct Eve Metrics sites.
What would be best is to stay as close as possible to existing data formats, so the existing applications can be re-used with minimal changes. What is needed is accurate data - those sites are somewhat notorious for occasionally having bad data in them. Actually... What would be best is providing an API so those sites can get accurate data and publish it via their already-exiting APIs, which the player base already uses.
As a side note, those site provide functionality for all Eve regions, with a 24-hour delay. I really don't see why CCP can't do the same. After all, those sites are written by players in their spare time, while CCP programmers are being paid to do this stuff.
|
|
Morar Santee
|
Posted - 2011.09.02 00:43:00 -
[191]
Originally by: CCP Dr.Eyjog Knowledge is power, and we want to give EVE Online pod pilots more power.
No **** bro? Then how about you give us the ability back to see character's standings? Just a thought.
|
hero honda
|
Posted - 2011.09.02 11:30:00 -
[192]
Originally by: Morar Santee
Originally by: CCP Dr.Eyjog Knowledge is power, and we want to give EVE Online pod pilots more power.
No **** bro? Then how about you give us the ability back to see character's standings? Just a thought.
Character sheet > standings.
|
Alice Cane
|
Posted - 2011.09.02 12:23:00 -
[193]
Is there no way possible that we can get just plain old OHCL data?
The open and close could be just before and after downtime.
If we could get that and feed it into our own charting programs, well.
That would be awesome.
|
DrTritanium Merc
|
Posted - 2011.09.03 01:10:00 -
[194]
Well they say nothing gets turned off. lol Right a few of my ships shields or armor stuff repairs or boosters were off line I had to play it out lucky in the field but okay if you think its fixed well?? Also going through most star gates the bubble goes backwards why?? let me know thanks. everyone else fly safe 0/
|
hero honda
|
Posted - 2011.09.03 04:09:00 -
[195]
Originally by: Alice Cane Is there no way possible that we can get just plain old OHCL data?
The open and close could be just before and after downtime.
If we could get that and feed it into our own charting programs, well.
That would be awesome.
Down time would be a nice time to compile a days worth of data, if there is time and space for it.
Besides I would not mind having candle stick charts for eve markets and you need the open and close for that.
|
Morar Santee
|
Posted - 2011.09.03 04:09:00 -
[196]
Edited by: Morar Santee on 03/09/2011 04:09:28
Originally by: hero honda
Originally by: Morar Santee
Originally by: CCP Dr.Eyjog Knowledge is power, and we want to give EVE Online pod pilots more power.
No **** bro? Then how about you give us the ability back to see character's standings? Just a thought.
Character sheet > standings.
Updated for "special" people.
|
hero honda
|
Posted - 2011.09.03 15:50:00 -
[197]
Originally by: Morar Santee Edited by: Morar Santee on 03/09/2011 04:09:28
Originally by: hero honda
Originally by: Morar Santee
Originally by: CCP Dr.Eyjog Knowledge is power, and we want to give EVE Online pod pilots more power.
No **** bro? Then how about you give us the ability back to see character's standings? Just a thought.
Character sheet > standings.
Updated for "special" people.
Updated? that is where is was in 2004 and still there.
|
Morar Santee
|
Posted - 2011.09.04 04:02:00 -
[198]
Seriously: Get a clue.
|
Evan183
Gallente EVE University Ivy League
|
Posted - 2011.09.06 01:42:00 -
[199]
I believe that a MySQL data backup (basically standard SQL with a few differences) would work best for many people, as MySQL is very, very popular.
Evan183 EVE University |
Dierdra Vaal
Caldari Veto. Veto Corp
|
Posted - 2011.09.06 13:02:00 -
[200]
Edited by: Dierdra Vaal on 06/09/2011 13:01:58 do we discuss here or on the new forums?
https://forums.eveonline.com/default.aspx?g=posts&t=6288
Veto #205 * * * Director Emeritus at EVE University * * * CSM1 delegate, CSM3 chairman and CSM5 vice-chairman
|
|
Diomedes Calypso
Aetolian Armada
|
Posted - 2011.09.08 16:31:00 -
[201]
Just to help me understand.
The figures will be fore average and median TRANSACTION prices I hope and absolutely ignore orders.
Orders are the price which no-one has yet sold or bought at, and are very frequently going to be innacurate both from manipulation and standard practice scam/oportunism market strategy (i.e shuttles for 8,888,888 , regional buys for 20 million implants at 1 isk etc).
Almost every item will have sell orders at 100 x the price just so people at the leading edge of the market will sometimes get a few hundred million windfall.
Simlar issues many items where the sellers maybe more patient than the buyers.. the buyers being retail customers, the seller's manufacturors .... I know from experience that the total volume of many items it far higher at one side of the bid ask spread than the other, and with wide bid ask spreads where that information is interesting.
tldr -- the information will be for actual transactions and ignore unexecuted buy and sell orders, won't it?
|
|
|
|
Pages: 1 2 3 4 5 6 7 :: [one page] |