Pages: [1] 2 3 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dr Slurm
General Commodities
|
Posted - 2006.12.28 19:13:00 -
[1]
The Market Data that I speak of is not the data that comes from the market logs. It comes from the table in "Price History" tab in the market window. By default this displays a graph, there's a button at the bottom that says show table.
This table gives you the data of what actually happened on a specific day. This is much more useful then the market logs that EVE generates. The market logs only present the orders that people have created. Although this does tell you a lot, just not all the history of the market.
Why I bring this up: I've made a couple of nice graphs that present the history of the market. These graphs are similar to those in game. The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. So far I've hand entered this data into a spreadsheet and converted it into a mysql table. This was very slow and tedious. So I found a software package that allows you to reliably pull data with only minor corrections needed.
What I need from other people: Screen shots of the above described table in consecutive order for at least 1 year time span for each mineral in each region. Screen shots should include no over-lapping data. Screen shots taken in the 1024x768 resolution seem to work best. There is also an option in the ESC settings menu to take high quality format screen shots. These are .BMP files and they do take up about 3.3mb each. Yes I know this is a lot but it helps the accuracy of the OCR software.
How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below).
What I will do: I will run the screen shots through this OCR software and weed out the bad data. And then compile it into a number of formats. (.CSV, EXCEL, MYSQL, or anything else reasonable)
What you get for helping: Everyone who helps gets a copy in their choice of formats of all data collected and access to daily updates. The public who did not contribute will have access to the data after 7 days, essentially receiving week old data.
I'm interested in all regions and all minerals for regions. So far from my testing it takes about 5 minutes to take screen shots of an entire year for one mineral.
I'd like feedback on this idea before I start an actual sign up and start collecting data.
|
Dr Slurm
General Commodities
|
Posted - 2006.12.28 19:14:00 -
[2]
Reserved for the moment.
|
SencneS
Amarr Keepers of the Holy Bagel The SUdden Death Squad
|
Posted - 2006.12.28 19:53:00 -
[3]
It's a great idea, but I can't help but think there may be an easier way.
http://eve-central.com/home/marketstat_xml.html?evemon=1
It grabs all the minerals, buy orders across all reporting regions across Eve.
These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space.
----------------------------------
Send ISK to SencneS for good Kama! |
Dr Slurm
General Commodities
|
Posted - 2006.12.28 20:11:00 -
[4]
Edited by: Dr Slurm on 28/12/2006 20:14:13
Originally by: SencneS It's a great idea, but I can't help but think there may be an easier way.
http://eve-central.com/home/marketstat_xml.html?evemon=1
It grabs all the minerals, buy orders across all reporting regions across Eve.
These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space.
That data is orders that ARE on the market currently. The data I want is what actually happened in the market for the day. The data is really only good for the previous data, but having a years worth on hand for each mineral in each region would be nice.
/edit: Here's a screenshot of what I mean.
|
Pang Grohl
Gallente
|
Posted - 2006.12.28 20:31:00 -
[5]
Any and every marketeer should be drooling over the opportunity this represents for OOG analysis of market trends in EVE.
I'll gladly volunteer my services to help you, Dr. Slurm. Sign me up for the Metropolis region for all minerals.
Also, what kind of deadline do you have for getting the data? And, could you let us in on the OCR software you're using? I need this kind of functionality for transaction logging from the wallet.
Si non adjuvas, noces (If you're not helping, you're hurting) |
Dr Slurm
General Commodities
|
Posted - 2006.12.28 20:39:00 -
[6]
I didn't mean to sound secretive, the software is ABBYY FineReader Pro 8.0 (FR). It's rather expensive and thus I didn't expect anyone else to do the OCR work.
So far this is my refining process: Take Screen shots move to individual folder like DOMAIN-ISO Open a batch of screenshots in FR Apply a table block format to all of them Adjust the table on each screen shot so it lines up Read all the pages Check each page for obvious errors like letters Remove all of " ; : | , . and really anything that isn't a number Save all the data to a .CSV file Open the .CSV file in a text editor Look over the data again to catch anymore mistakes and remove any "'s that are generated by FR Save any changes Insert .CSV into Open Office to check for more errors
This pretty much cleans the data up. I'll admit there's a few errors but that is in part thanks to the eve client. This does leave the ISK values for LOW HIGH and AVG without proper decimnal places but this can be fixed by dividing all data by 100.
|
Ramblin Man
Empyreum
|
Posted - 2006.12.28 20:52:00 -
[7]
Edited by: Ramblin Man on 28/12/2006 20:52:16
I'll help out. (Will check later when I've got time and see what region/data is unclaimed)
Where's the "window" by default in the OCR software? Or do you have to manually draw it? It seems like a bit of time could be saved by having people properly position their market windows ingame before taking screenshots.
|
Dr Slurm
General Commodities
|
Posted - 2006.12.28 20:55:00 -
[8]
You can see an example screenshot in one of my above posts.
I doubt everyone will be able to get it exactly the same.
I will update the first post with screen shot requirements.
|
Technovar
|
Posted - 2006.12.29 22:59:00 -
[9]
Originally by: Dr Slurm
Images should be placed in a folder similar to their nature i.e. JITA-TRIT. All folders containing images should be archived (zip, rar, gz I don't care) Email all files to [email protected]
How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below).
*snip*
Originally by: Dr Slurm
I'd like feedback on this idea before I start an actual sign up and start collecting data.
I know you're looking for feedback, but it does look like you've gotten good feedback so far, so I took action.
Since I'm reasonably new, my region choice is somewhat limited if I want to stay alive. So, I've got all minerals in Verge Vendor screenshotted from 2005-Nov-01 to 2006-Dec-28, rar'ed into a 75MB file, and ready to go when you do take signups. If you have available space in that account, I can go ahead and mail it. I'm certainly interested in the SQL once you get this done, as well.
|
Dr Slurm
General Commodities
|
Posted - 2006.12.29 23:15:00 -
[10]
I guess the full sign up is on. I can't think of anymore kinks to work out.
I've already finished the OCR work for Domain and Essence. I have Heimatar already screen shotted. You are welcome to send what you have [email protected] . The email box is 1gig so I should have plenty of room.
Please I encourage more people to be a part of this. It doesn't take more then 2-3 mins to grab the screen shots of one mineral. It might take 20 mins to grab all of them for one region. This isn't hard and you reap large amount of otherwise unattainable data from it. Check out the Market History OCR Project here |
|
Dr Slurm
General Commodities
|
Posted - 2006.12.30 16:43:00 -
[11]
I had to change the email to [email protected] hopefully there won't be virus scanning issues like with hotmail.
I also updated the first post to say that screenshots need to be taken on a 0% transparency background preferably solid black. Check out the Market History OCR Project here |
Ray McCormack
BIG
|
Posted - 2006.12.30 18:39:00 -
[12]
I was pretty sure The Forge wasn't marked when I started doing it... Oh well.
| How To Afford A Tech II BPO | the all mighty BIG industrial corp from hell with a slong the size of a walrus. |
Dr Slurm
General Commodities
|
Posted - 2006.12.30 18:45:00 -
[13]
I'm sorry, I hate to waste your time, I just finished it. Please let me know if you'd be willing to do one of the other priority regions before you start.
Check out the Market History OCR Project here |
Ki Shodan
Gallente Lunar Dawn Storm Armada
|
Posted - 2006.12.30 20:58:00 -
[14]
I am taking shots of all minerals in "Devoid", "The Bleak Lands" and "Kador" regions, if still available. (Can do regular updates on Domain later, too.) Is Morphite also needed or just the normal minerals? -- Ki Shodans Sweat Shop
Evemail me, if my name is used as guarantor! |
Dr Slurm
General Commodities
|
Posted - 2006.12.30 21:13:00 -
[15]
I will sign you up.
Just the normal minerals Trit, Pye, Mex, Iso, Noc, Zyd, and Mega. Check out the Market History OCR Project here |
Menkaure
Amarr Vanitas Corp.
|
Posted - 2006.12.30 23:45:00 -
[16]
Guess I can take lonetrek. :)
|
Dr Slurm
General Commodities
|
Posted - 2006.12.31 01:22:00 -
[17]
I'm making good head way processing the screen shots I've taken myself and those sent to me by Technovar. I've refined the process and am able to check data much quicker now.
If I had all the screen shots tonight I could come close to finishing. I can take them, but I'd rather spend my time processing. Check out the Market History OCR Project here |
Menkaure
Amarr Vanitas Corp.
|
Posted - 2006.12.31 03:57:00 -
[18]
Got them for you but Hotmail/Gmail both seem to be crapping out when I try to upload em so I'll try again tommorow morning at some point.
|
Dr Slurm
General Commodities
|
Posted - 2006.12.31 04:13:00 -
[19]
Technovar was able to send them through google in about 9mb packages. One archive for each mineral. Check out the Market History OCR Project here |
Menkaure
Amarr Vanitas Corp.
|
Posted - 2006.12.31 16:14:00 -
[20]
k you should have mine now, ill hop over to the citadel and do them aswell
|
|
Dr Slurm
General Commodities
|
Posted - 2007.01.01 07:17:00 -
[21]
Edited by: Dr Slurm on 01/01/2007 07:25:05 I currently have all of The Forge, Domain, Essence, Heimatar, and Verge Vendor OCR'ed and purified of erroneous data.
I currently have 2 regions and some change to work on.
Just an update.
/I put my name on The Citadel by accident. I hope that didn't confuse.
Tip to contributors: If you expand the size of the column in the eve market window it gives me more wiggle room to place the template the OCR software uses over it. Check out the Market History OCR Project here |
Pang Grohl
Gallente
|
Posted - 2007.01.01 09:11:00 -
[22]
Dr. did you get the screenies I sent you for Metropolis?
Si non adjuvas, noces (If you're not helping, you're hurting) |
Dr Slurm
General Commodities
|
Posted - 2007.01.01 12:07:00 -
[23]
Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis?
Did you send it to [email protected] ? the hotmail address wouldn't accept the large attachments.
Either way I haven't found an email saying it was from you. Check out the Market History OCR Project here |
Pang Grohl
Gallente
|
Posted - 2007.01.02 16:30:00 -
[24]
Originally by: Dr Slurm
Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis?
Did you send it to [email protected] ? the hotmail address wouldn't accept the large attachments.
Either way I haven't found an email saying it was from you.
I did send it to the gmail address. I had some contradictory messages from my e-mail client about the success of the send & wanted to make sure you didn't get it before I resent. I will try again now.
Si non adjuvas, noces (If you're not helping, you're hurting) |
Dr Slurm
General Commodities
|
Posted - 2007.01.03 13:18:00 -
[25]
I still haven't received an email from Pang.
I really appreciate everyone's help so far, this is going to be a great project.
Check out the Market History OCR Project here |
Jade Vendetta
Gallente Cospoli Assembly
|
Posted - 2007.01.03 15:48:00 -
[26]
Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.
|
Dr Slurm
General Commodities
|
Posted - 2007.01.03 23:53:00 -
[27]
Originally by: Jade Vendetta Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.
You have been signed up.
Only 9 real regions left for the priority group, 10 if you count the unpronounceable one. Check out the Market History OCR Project here |
Menkaure
Amarr Vanitas Corp.
|
Posted - 2007.01.04 01:47:00 -
[28]
Are we going to have to wait till those 10 regions are in before you show the data?
|
Icy One
|
Posted - 2007.01.04 09:57:00 -
[29]
I can take on doing all minerals in the Tash-Murkon region. I will get some SS to you after the extended DT.
|
Dr Slurm
General Commodities
|
Posted - 2007.01.05 12:47:00 -
[30]
Originally by: Menkaure Are we going to have to wait till those 10 regions are in before you show the data?
I'd rather not redistribute a bunch of different copies. I've setup an SVN (subversion) on my local computer. I'm not sure if it's accessible from the outside. Once I get that setup all that contributed will be welcome to check out and possibly commit changes.
Check out the Market History OCR Project here |
|
|
|
|
Pages: [1] 2 3 4 :: one page |
First page | Previous page | Next page | Last page |