EVE Search - Market Data OCR Project

Check My IP Information

All Channels

Market Discussions

Market Data OCR Project

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 3 4 :: [one page]

Author	Thread Statistics \| Show CCP posts - 0 post(s)
Dr Slurm General Commodities	Posted - 2006.12.28 19:13:00 - [1] The Market Data that I speak of is not the data that comes from the market logs. It comes from the table in "Price History" tab in the market window. By default this displays a graph, there's a button at the bottom that says show table. This table gives you the data of what actually happened on a specific day. This is much more useful then the market logs that EVE generates. The market logs only present the orders that people have created. Although this does tell you a lot, just not all the history of the market. Why I bring this up: I've made a couple of nice graphs that present the history of the market. These graphs are similar to those in game. The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. So far I've hand entered this data into a spreadsheet and converted it into a mysql table. This was very slow and tedious. So I found a software package that allows you to reliably pull data with only minor corrections needed. What I need from other people: Screen shots of the above described table in consecutive order for at least 1 year time span for each mineral in each region. Screen shots should include no over-lapping data. Screen shots taken in the 1024x768 resolution seem to work best. There is also an option in the ESC settings menu to take high quality format screen shots. These are .BMP files and they do take up about 3.3mb each. Yes I know this is a lot but it helps the accuracy of the OCR software. How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below). What I will do: I will run the screen shots through this OCR software and weed out the bad data. And then compile it into a number of formats. (.CSV, EXCEL, MYSQL, or anything else reasonable) What you get for helping: Everyone who helps gets a copy in their choice of formats of all data collected and access to daily updates. The public who did not contribute will have access to the data after 7 days, essentially receiving week old data. I'm interested in all regions and all minerals for regions. So far from my testing it takes about 5 minutes to take screen shots of an entire year for one mineral. I'd like feedback on this idea before I start an actual sign up and start collecting data.
Dr Slurm General Commodities	Posted - 2006.12.28 19:14:00 - [2] Reserved for the moment.
SencneS Amarr Keepers of the Holy Bagel The SUdden Death Squad	Posted - 2006.12.28 19:53:00 - [3] It's a great idea, but I can't help but think there may be an easier way. http://eve-central.com/home/marketstat_xml.html?evemon=1 It grabs all the minerals, buy orders across all reporting regions across Eve. These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space. ---------------------------------- Send ISK to SencneS for good Kama!
Dr Slurm General Commodities	Posted - 2006.12.28 20:11:00 - [4] Edited by: Dr Slurm on 28/12/2006 20:14:13 Originally by: SencneS It's a great idea, but I can't help but think there may be an easier way. http://eve-central.com/home/marketstat_xml.html?evemon=1 It grabs all the minerals, buy orders across all reporting regions across Eve. These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space. That data is orders that ARE on the market currently. The data I want is what actually happened in the market for the day. The data is really only good for the previous data, but having a years worth on hand for each mineral in each region would be nice. /edit: Here's a screenshot of what I mean.
Pang Grohl Gallente	Posted - 2006.12.28 20:31:00 - [5] Any and every marketeer should be drooling over the opportunity this represents for OOG analysis of market trends in EVE. I'll gladly volunteer my services to help you, Dr. Slurm. Sign me up for the Metropolis region for all minerals. Also, what kind of deadline do you have for getting the data? And, could you let us in on the OCR software you're using? I need this kind of functionality for transaction logging from the wallet. Si non adjuvas, noces (If you're not helping, you're hurting)
Dr Slurm General Commodities	Posted - 2006.12.28 20:39:00 - [6] I didn't mean to sound secretive, the software is ABBYY FineReader Pro 8.0 (FR). It's rather expensive and thus I didn't expect anyone else to do the OCR work. So far this is my refining process: Take Screen shots move to individual folder like DOMAIN-ISO Open a batch of screenshots in FR Apply a table block format to all of them Adjust the table on each screen shot so it lines up Read all the pages Check each page for obvious errors like letters Remove all of " ; : \| , . and really anything that isn't a number Save all the data to a .CSV file Open the .CSV file in a text editor Look over the data again to catch anymore mistakes and remove any "'s that are generated by FR Save any changes Insert .CSV into Open Office to check for more errors This pretty much cleans the data up. I'll admit there's a few errors but that is in part thanks to the eve client. This does leave the ISK values for LOW HIGH and AVG without proper decimnal places but this can be fixed by dividing all data by 100.
Ramblin Man Empyreum	Posted - 2006.12.28 20:52:00 - [7] Edited by: Ramblin Man on 28/12/2006 20:52:16 I'll help out. (Will check later when I've got time and see what region/data is unclaimed) Where's the "window" by default in the OCR software? Or do you have to manually draw it? It seems like a bit of time could be saved by having people properly position their market windows ingame before taking screenshots.
Dr Slurm General Commodities	Posted - 2006.12.28 20:55:00 - [8] You can see an example screenshot in one of my above posts. I doubt everyone will be able to get it exactly the same. I will update the first post with screen shot requirements.
Technovar	Posted - 2006.12.29 22:59:00 - [9] Originally by: Dr Slurm Images should be placed in a folder similar to their nature i.e. JITA-TRIT. All folders containing images should be archived (zip, rar, gz I don't care) Email all files to [email protected] How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below). snip Originally by: Dr Slurm I'd like feedback on this idea before I start an actual sign up and start collecting data. I know you're looking for feedback, but it does look like you've gotten good feedback so far, so I took action. Since I'm reasonably new, my region choice is somewhat limited if I want to stay alive. So, I've got all minerals in Verge Vendor screenshotted from 2005-Nov-01 to 2006-Dec-28, rar'ed into a 75MB file, and ready to go when you do take signups. If you have available space in that account, I can go ahead and mail it. I'm certainly interested in the SQL once you get this done, as well.
Dr Slurm General Commodities	Posted - 2006.12.29 23:15:00 - [10] I guess the full sign up is on. I can't think of anymore kinks to work out. I've already finished the OCR work for Domain and Essence. I have Heimatar already screen shotted. You are welcome to send what you have [email protected] . The email box is 1gig so I should have plenty of room. Please I encourage more people to be a part of this. It doesn't take more then 2-3 mins to grab the screen shots of one mineral. It might take 20 mins to grab all of them for one region. This isn't hard and you reap large amount of otherwise unattainable data from it. Check out the Market History OCR Project here

Dr Slurm General Commodities	Posted - 2006.12.30 16:43:00 - [11] I had to change the email to [email protected] hopefully there won't be virus scanning issues like with hotmail. I also updated the first post to say that screenshots need to be taken on a 0% transparency background preferably solid black. Check out the Market History OCR Project here
Ray McCormack BIG	Posted - 2006.12.30 18:39:00 - [12] I was pretty sure The Forge wasn't marked when I started doing it... Oh well. \| How To Afford A Tech II BPO \| the all mighty BIG industrial corp from hell with a slong the size of a walrus.
Dr Slurm General Commodities	Posted - 2006.12.30 18:45:00 - [13] I'm sorry, I hate to waste your time, I just finished it. Please let me know if you'd be willing to do one of the other priority regions before you start. Check out the Market History OCR Project here
Ki Shodan Gallente Lunar Dawn Storm Armada	Posted - 2006.12.30 20:58:00 - [14] I am taking shots of all minerals in "Devoid", "The Bleak Lands" and "Kador" regions, if still available. (Can do regular updates on Domain later, too.) Is Morphite also needed or just the normal minerals? -- Ki Shodans Sweat Shop Evemail me, if my name is used as guarantor!
Dr Slurm General Commodities	Posted - 2006.12.30 21:13:00 - [15] I will sign you up. Just the normal minerals Trit, Pye, Mex, Iso, Noc, Zyd, and Mega. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2006.12.30 23:45:00 - [16] Guess I can take lonetrek. :)
Dr Slurm General Commodities	Posted - 2006.12.31 01:22:00 - [17] I'm making good head way processing the screen shots I've taken myself and those sent to me by Technovar. I've refined the process and am able to check data much quicker now. If I had all the screen shots tonight I could come close to finishing. I can take them, but I'd rather spend my time processing. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2006.12.31 03:57:00 - [18] Got them for you but Hotmail/Gmail both seem to be crapping out when I try to upload em so I'll try again tommorow morning at some point.
Dr Slurm General Commodities	Posted - 2006.12.31 04:13:00 - [19] Technovar was able to send them through google in about 9mb packages. One archive for each mineral. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2006.12.31 16:14:00 - [20] k you should have mine now, ill hop over to the citadel and do them aswell

Dr Slurm General Commodities	Posted - 2007.01.01 07:17:00 - [21] Edited by: Dr Slurm on 01/01/2007 07:25:05 I currently have all of The Forge, Domain, Essence, Heimatar, and Verge Vendor OCR'ed and purified of erroneous data. I currently have 2 regions and some change to work on. Just an update. /I put my name on The Citadel by accident. I hope that didn't confuse. Tip to contributors: If you expand the size of the column in the eve market window it gives me more wiggle room to place the template the OCR software uses over it. Check out the Market History OCR Project here
Pang Grohl Gallente	Posted - 2007.01.01 09:11:00 - [22] Dr. did you get the screenies I sent you for Metropolis? Si non adjuvas, noces (If you're not helping, you're hurting)
Dr Slurm General Commodities	Posted - 2007.01.01 12:07:00 - [23] Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis? Did you send it to [email protected] ? the hotmail address wouldn't accept the large attachments. Either way I haven't found an email saying it was from you. Check out the Market History OCR Project here
Pang Grohl Gallente	Posted - 2007.01.02 16:30:00 - [24] Originally by: Dr Slurm Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis? Did you send it to [email protected] ? the hotmail address wouldn't accept the large attachments. Either way I haven't found an email saying it was from you. I did send it to the gmail address. I had some contradictory messages from my e-mail client about the success of the send & wanted to make sure you didn't get it before I resent. I will try again now. Si non adjuvas, noces (If you're not helping, you're hurting)
Dr Slurm General Commodities	Posted - 2007.01.03 13:18:00 - [25] I still haven't received an email from Pang. I really appreciate everyone's help so far, this is going to be a great project. Check out the Market History OCR Project here
Jade Vendetta Gallente Cospoli Assembly	Posted - 2007.01.03 15:48:00 - [26] Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.
Dr Slurm General Commodities	Posted - 2007.01.03 23:53:00 - [27] Originally by: Jade Vendetta Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with. You have been signed up. Only 9 real regions left for the priority group, 10 if you count the unpronounceable one. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.04 01:47:00 - [28] Are we going to have to wait till those 10 regions are in before you show the data?
Icy One	Posted - 2007.01.04 09:57:00 - [29] I can take on doing all minerals in the Tash-Murkon region. I will get some SS to you after the extended DT.
Dr Slurm General Commodities	Posted - 2007.01.05 12:47:00 - [30] Originally by: Menkaure Are we going to have to wait till those 10 regions are in before you show the data? I'd rather not redistribute a bunch of different copies. I've setup an SVN (subversion) on my local computer. I'm not sure if it's accessible from the outside. Once I get that setup all that contributed will be welcome to check out and possibly commit changes. Check out the Market History OCR Project here

Block Ukx	Posted - 2007.01.05 19:22:00 - [31] Sing me up for Khanid, if it's available.
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.06 02:09:00 - [32] Oh, and why is J7HZ-F listed? Its Jovian and inacessible.... Might be tough to get screenies.
Dr Slurm General Commodities	Posted - 2007.01.06 05:16:00 - [33] Originally by: Menkaure Oh, and why is J7HZ-F listed? Its Jovian and inacessible.... Might be tough to get screenies. Sorry, updated. I just took a query from the mapRegions table of any region with a solar system with of .5 or greater security. Anything else ended up in the non-priority list. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.07 18:18:00 - [34] Here is an update to the available formats for those who contributed: CSV CSV for MS Excel Microsoft Excel 2000 Microsoft Word 2000 LaTeX Open Document Spreadsheet Open Document Text PDF SQL XML These are all generated by phpMyAdmin. I have 5 regions with all minerals in a mysql database now. I'm still working on processing the last two I currently have. I will probably run out of regions to work on tonight. So if you have some screen shots send them to my gmail account. Check out the Market History OCR Project here
Splagada Minmatar Tides of Silence	Posted - 2007.01.08 13:39:00 - [35] it would be against the terms of service, but a simple macro that goes from top to bottom in browse tab, reads the color of the screen (to know if you're in a category tab or not), or simply stupidly goes, knowing the tree structure, and click export to file this and the eve central market auto uploader and your entire region, all items is done in a few minutes its very borderline with the code of conduct tho. its out of game advantage mostly, i dotn know lol ------ relaxed corp looking for members
Dr Slurm General Commodities	Posted - 2007.01.08 22:50:00 - [36] Originally by: Splagada it would be against the terms of service, but a simple macro that goes from top to bottom in browse tab, reads the color of the screen (to know if you're in a category tab or not), or simply stupidly goes, knowing the tree structure, and click export to file this and the eve central market auto uploader and your entire region, all items is done in a few minutes its very borderline with the code of conduct tho. its out of game advantage mostly, i dotn know lol The data I'm collecting is not what you are thinking of. The data that is exported by EVE are the orders currently on the market. These orders can be filled/canceled/changed at the will of the owners. What I'm collecting is the history of market orders for each region. If you look on the first page I think I explained this and gave an example screenshot. The market history provides the date, total number of orders, total volume, and low, high, and average prices for the day. This market history is actually useful where as the market orders data is really rather quite useless. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.11 14:38:00 - [37] I have started a corporation in regards to this project. It can be found here. Anyone who contributed to this project will be given shares accordingly. These shares were paid for with your "volunteer" work. No ISK required. I really hope more people take advantage of this opportunity. Update: The conversion of data is going well, there are still a few regions left that I haven't had anyone interested in. I will probably end doing these myself, but this does take away time from purifying the data. 1 Priority region = 500 Shares in Eve Data Cooperative If you think about any other IPO comparing the time it takes you to make the ISK to buy the shares vs the time it takes to take screen shots and email them to me this is a considerably good deal. Check out the Market History OCR Project here
Lee Walker	Posted - 2007.01.11 16:01:00 - [38] I just have one question: In order to take a screenshot people have to wait for the buy and sell orders to be fetched from the server, then they have to take the screenshot, zip it up, and send it to you. In addition they have to change the transparency take care of window positioning etc. Why not just have them click the export button and mail you the EVE export? I just don't get it. The OCR project seems to require more work on the part of those who submit the data, and certainly requires a lot more work on your part in terms of data QA/QC. Don't get me wrong, great that you are putting in effort, it just seems like the method wastes a lot of everyone's time. Lee Walker speaks out as to what really happened to EVE-Trader.net
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.11 17:31:00 - [39] Screw it, i'll do Kor-Azor aswell; but mostly because I'm impatient to get a hold of the data. Might do a few others tonight depending on how bored I get.
Block Ukx	Posted - 2007.01.11 17:41:00 - [40] OK, sign me up for Molden Heath. Dr. Slurm, Was the all-minerlas Metropolis data I sent you acceptable? Please let me know before I start this new region. I don't want to repeat a misstake.

Menkaure Amarr Vanitas Corp.	Posted - 2007.01.11 17:51:00 - [41] Originally by: Dr Slurm Edited by: Dr Slurm on 07/01/2007 14:36:03 Originally by: Menkaure Oh, and why is J7HZ-F listed? Its Jovian and inacessible.... Might be tough to get screenies. Sorry, updated. I just took a query from the mapRegions table of any region with a solar system of .5 or greater security. Anything else ended up in the non-priority list. On that note aswell, you may wanna take out UUA-F4 and A821-A as they're both inaccessible aswell.
Dak Biet	Posted - 2007.01.11 21:17:00 - [42] I'd like to sign up. I currently operate in Lonetrek and the Citadel, but I can park an alt somewhere else for the screenshots. I'll pick a spot tonight.
Dr Slurm General Commodities	Posted - 2007.01.11 23:06:00 - [43] Edited by: Dr Slurm on 11/01/2007 23:09:55 Originally by: Menkaure Screw it, i'll do Kor-Azor aswell; but mostly because I'm impatient to get a hold of the data. Might do a few others tonight depending on how bored I get. Thank you Originally by: Block Ukx OK, sign me up for Molden Heath. Dr. Slurm, Was the all-minerlas Metropolis data I sent you acceptable? Please let me know before I start this new region. I don't want to repeat a misstake. Please just name your folders and zip files like molden-iso. Otherwise the screenshots were dead on, and I believe you even took the extra effort to edit out some bad data. Originally by: Dak Biet I'd like to sign up. I currently operate in Lonetrek and the Citadel, but I can park an alt somewhere else for the screenshots. I'll pick a spot tonight. Just let me know what you want, the list on the first page has been updated. Check out the Market History OCR Project here
Pang Grohl Gallente	Posted - 2007.01.11 23:45:00 - [44] Oy! Gmail and/or yahoo must hate me Since it appears you don't need the data on Metro anymore, I'll be glad to grab Genesis for you. Si non adjuvas, noces (If you're not helping, you're hurting)
Dr Slurm General Commodities	Posted - 2007.01.11 23:57:00 - [45] Originally by: Pang Grohl Oy! Gmail and/or yahoo must hate me Since it appears you don't need the data on Metro anymore, I'll be glad to grab Genesis for you. I'm not sure what you mean. I received your data and entered it. I'll gladly sign you up for gensis though. Check out the Market History OCR Project here
Dak Biet	Posted - 2007.01.12 00:15:00 - [46] Originally by: Dr Slurm Edited by: Dr Slurm on 11/01/2007 23:09:55 Originally by: Menkaure Screw it, i'll do Kor-Azor aswell; but mostly because I'm impatient to get a hold of the data. Might do a few others tonight depending on how bored I get. Thank you Originally by: Block Ukx OK, sign me up for Molden Heath. Dr. Slurm, Was the all-minerlas Metropolis data I sent you acceptable? Please let me know before I start this new region. I don't want to repeat a misstake. Please just name your folders and zip files like molden-iso. Otherwise the screenshots were dead on, and I believe you even took the extra effort to edit out some bad data. Originally by: Dak Biet I'd like to sign up. I currently operate in Lonetrek and the Citadel, but I can park an alt somewhere else for the screenshots. I'll pick a spot tonight. Just let me know what you want, the list on the first page has been updated. What the heck, I'll do Aridia. Sign me up please!
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.12 00:21:00 - [47] Edited by: Menkaure on 12/01/2007 00:22:23 Made a disposable alt that i thought i might try and get data for some random 0.0 region. Thought it might be fun. If there's anything wrong with my screenies then speak now cus i (might) be doing more. You should have my tash's now aswell.
Pang Grohl Gallente	Posted - 2007.01.12 00:22:00 - [48] Originally by: Dr Slurm Originally by: Pang Grohl Oy! Gmail and/or yahoo must hate me Since it appears you don't need the data on Metro anymore, I'll be glad to grab Genesis for you. I'm not sure what you mean. I received your data and entered it. I'll gladly sign you up for gensis though. Ah, good news. I saw Block's post about the Metro data & assumed you hadn't received my second attempt (silly me, assuming). I will be in Genesis tonight & will capture that data. Si non adjuvas, noces (If you're not helping, you're hurting)
Dr Slurm General Commodities	Posted - 2007.01.12 00:27:00 - [49] Dak Biet you have Aridia. I just delisted Placid and Solitude from the Priority list to the non-priority. If anyone was planning on doing these I'm sorry. They are still worth 250 shares each though. That leaves Derelik and Everyshore. I have yet to receive Tash-Murkon from Icy One. If you're holding out please send it. I'm very close to finishing all processing. If I don't receive it in a day I will put it back up for sign up or do it myself. P.S. Jade Vendetta your screen shots were so perfect I didn't have to edit the template used in the OCR process. Which made it very easy and quick to process sing liason. After I release this data I expect to find errors in it. Errors will be removed and somehow distributed to the members. I was wondering if everyone would be willing to do spot audits on the data that they submitted. This would consist of comparing the screen shots to the data and sending any corrections to me. Please keep in mind I've done my absolute best to make this data error free, but I'm not perfect and neither is the OCR software. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.12 00:33:00 - [50] Originally by: Menkaure Edited by: Menkaure on 12/01/2007 00:25:32 Made a disposable alt that i thought i might try and get data for some random 0.0 region. Thought it might be fun. If there's anything wrong with my screenies then speak now cus i (might) be doing more. You should have my kor azor's now aswell. (edit, put tash instead :S) I've received your Kor Azors. Your screenies are great, please contribute more. A little tip for those still shooting if you sort the table by date Ascending it makes it easier to take the shots. That way you can just start at 12/01/05 and not overlap any data. Thanks guys, this is turning out to be a great project and I really don't think I could have brought myself to do it all. And definitely not as fast, this has been a great response. Check out the Market History OCR Project here

Menkaure Amarr Vanitas Corp.	Posted - 2007.01.12 02:16:00 - [51] Edited by: Menkaure on 12/01/2007 02:15:43 Providence, Catch and Stain coming your way. I feel I might be taking this far too seriously. 6 Regions and counting
Block Ukx	Posted - 2007.01.12 02:56:00 - [52] Just to clarify, I scanned Khanid, not Metropolis. I'm sorry for the confussion my earlier post might have caused. I corrected my previous post
Dak Biet	Posted - 2007.01.13 03:40:00 - [53] Originally by: Dr Slurm That leaves Derelik and Everyshore. Sign me up for Everyshore too...
Dr Slurm General Commodities	Posted - 2007.01.13 04:36:00 - [54] Done Check out the Market History OCR Project here
Feya Kortimna	Posted - 2007.01.13 16:04:00 - [55] Derelik for me, please.
Dr Slurm General Commodities	Posted - 2007.01.13 16:48:00 - [56] Originally by: Feya Kortimna Derelik for me, please. And that, kids, is the last of the Priority Regions. I've updated the second post with the status of each region. I've yet to have any contact with Icy One. After I have received and processed the Priority Regions I will be releasing the data to the Members. Check out the Market History OCR Project here
Feya Kortimna	Posted - 2007.01.13 17:10:00 - [57] Originally by: Feya Kortimna Derelik for me, please. Should now be in your inbox now. GMail mistakenly claimed that there was an executable in the file when it was in .zip format, so now it's in 7z format.
Dr Slurm General Commodities	Posted - 2007.01.13 17:27:00 - [58] Thanks, I got it. I just created a SVN on http://opensvn.csie.org/. If I have one fully processed region from you I sent you a login and password, which you can change, to the email that you contacted me with (Jade Vendetta if you didn't get yours email me again). A lot of other websites could explain what SVN is better then I, but let me just say its a version control system that tracks changes made to text files. The way I plan to use this is to distribute SQL and CSV files for editing purposes. If you find an error in the files vs the screen shots you can then correct the mistake and it will later be distributed to all. To use svn you will need TortoiseSVN. It's basically a plugin for windows explorer. It integrates with it directly. Simple Setup: Install TortoiseSVN Create folder anywhere name it something clever like "EveDataCooperative" right click the folder, TortoiseSVN -> Create repository here... It will ask your a for url of where to find the SVN, enter this: http://opensvn.csie.org/EveDataCooperative It will then ask for a username and password, use the login I sent you(unless you changed it). Hit OK Right click the folder again, choose SVN checkout. This will download everything contained within the SVN. You should now have a copy of everything in the SVN repository. Advanced: You should probably read up on SVN if you want to do any editing or committing of data. Here is very good online book detailing SVN. If you are interested in editing, please read. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.18 17:01:00 - [59] Pang: If you see this check your email please, there was a problem with your screen shots. thnx, Dr Slurm Check out the Market History OCR Project here
Pang Grohl Gallente	Posted - 2007.01.19 01:11:00 - [60] Originally by: Dr Slurm Pang: If you see this check your email please, there was a problem with your screen shots. thnx, Dr Slurm I will check as soon as I get home from work. Hopefully it's an easily correctable problem. Si non adjuvas, noces (If you're not helping, you're hurting)

Menkaure Amarr Vanitas Corp.	Posted - 2007.01.20 07:00:00 - [61] If Icy One hasn't got you tash within a week or so I'll swing round and get it. Just gimme a bell ingame or on here or somesuch.
Dr Slurm General Commodities	Posted - 2007.01.22 02:00:00 - [62] Tash-Murkon is back up for grabs and the 500 shares that go with it. The next person to respond with interest gets it. (Just to be fair Menk) Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.22 13:16:00 - [63] Edited by: Dr Slurm on 22/01/2007 13:18:16 Region # Errors # Entries Ratio % Derelik 24028700.08 The Forge 6127300.02 Lonetrek 7929800.03 Heimatar 21427320.08 The Citadel 7429880.02 Devoid 12527990.04 The Bleak Lands 12428520.04 Metropolis 7929820.03 Domain 9227300.03 Khanid 10128210.04 Kador 4428410.02 Essence 22526230.09 Kor-Azor 27728180.1 Verge Vendor 9929750.03 SUM1834397410.05 This is a summary from the database that I pulled. It shows the number of errors that I know exist for sure. This was just a simple check to make sure the low, high and avg related to each other properly. More errors then this probably exist. The last line is the total. Slurm Check out the Market History OCR Project here
Feya Kortimna	Posted - 2007.01.25 05:21:00 - [64] Do you have some 'automagical' error-correction in mind? I'll be happy to poke around in the screenshots / data I provided to clean up those errors, if the solution is manual checking. If you send me a TortoiseSVN userid, I will be able to get at the error-rich data, right?
Dr Slurm General Commodities	Posted - 2007.01.25 05:51:00 - [65] Originally by: Feya Kortimna Do you have some 'automagical' error-correction in mind? I'll be happy to poke around in the screenshots / data I provided to clean up those errors, if the solution is manual checking. If you send me a TortoiseSVN userid, I will be able to get at the error-rich data, right? Well the regions with an error rate higher then 3% I was just going to do again since its not that hard. Just a little time consuming. The others I was just going to look through to see if it as something obvious. I have to say for beginning run in an OCR project %5 error ratio is pretty decent, but then again I'm sure there are errors that I couldn't easily check for. I will check in all my data and send you a login. Sidurii notified me ingame that they will be doing the Placid region and Tash-Murkon. As for an update, I'm finishing up Genesis right now, and as soon as I receive Tash I will process it. The non-priority regions will wait until the Priority regions are fully processed and published for the membership. I hope to make my self imposed deadline of the end of the month, but I worry there will be a lot errors in the data, devaluing its worth. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.25 07:04:00 - [66] I just checked in a .sql and a .csv dump of the data into the SVN repository. It's pretty easy to use, check it out. I'd link it but I'm tired and you all should have a received an email from opensvn.csie.org with a login and password. Enjoy! I'll be updating it as I and others correct errors. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.27 18:12:00 - [67] Edited by: Menkaure on 27/01/2007 18:10:29 Originally by: Dr Slurm I just checked in a .sql and a .csv dump of the data into the SVN repository. It's pretty easy to use, check it out. I'd link it but I'm tired and you all should have a received an email from opensvn.csie.org with a login and password. Enjoy! I'll be updating it as I and others correct errors. One small quibble, had a look at the data but can you chance the CSV so that the data is split by something shorter than "& q u o t ; ; & q u o t ;" ?? Like, just a comma or ;; or something? (heh, forum kept on eating it so put loadsa spaces in to split it up. Spaces aint present in the actual CSV. (when i put in what it actually is, it comes out as ";" ) Thats too long for openoffice calc to split it into columns (dunno about excel) and I don't particularly wanna kill my computer for a few hours by replacing it in notepad or something. (yes, if there's an easier way to do it then tell me- I am a newb at this kinda stuff)
Motivated Prophet Zerodot Schools Power Corrupts Industry's	Posted - 2007.01.27 18:28:00 - [68] I believe Wordpad (write.exe in your "Run" box) will not update the screen until after the replacement operation is complete, which means it won't take hours, just seconds. Of course, if the entire thing is HTML-encoded, just stick "<PRE>" at the start (sans quotes), save it to a file, change the file's extension to .HTM, open it in a browser, and copy and paste the data back where you need it. MP
Kua Burrow	Posted - 2007.01.27 19:57:00 - [69] Originally by: Dr Slurm The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. Um this what you're doing sounds interesting and useful but at the risk of sounding dumb I'm not really sure what it is you are actually doing . The above quote seems to be the crux of the matter - would someone expand on it and explain in layman's terms what this project involves and (more importantly) what it will leave us with that will mean we are better off than b4? Thanks.
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.27 20:04:00 - [70] Originally by: Motivated Prophet I believe Wordpad (write.exe in your "Run" box) will not update the screen until after the replacement operation is complete, which means it won't take hours, just seconds. Of course, if the entire thing is HTML-encoded, just stick "<PRE>" at the start (sans quotes), save it to a file, change the file's extension to .HTM, open it in a browser, and copy and paste the data back where you need it. MP Notepad I left for about 2 hours trying to do it before i closed it down, but you're right, wordpad is better. Managed it after about half an hour. Thanks for the tip.

Dr Slurm General Commodities	Posted - 2007.01.28 00:33:00 - [71] Wow I didn't even look at what it put out. I'll fix it and check in the changes Check out the Market History OCR Project here
gazarsgo Einherjar Rising	Posted - 2007.01.30 15:35:00 - [72] from the 1.3 notes: Quote: + Market History graphs should now display the correct values of ISK instead of cropping the most significant digit. Does this affect your work to date at all?
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.30 15:58:00 - [73] Originally by: gazarsgo from the 1.3 notes: Quote: + Market History graphs should now display the correct values of ISK instead of cropping the most significant digit. Does this affect your work to date at all? No that refers to the graph, this project uses the table.
Dr Slurm General Commodities	Posted - 2007.01.31 14:01:00 - [74] I did a massive replace to the mysql database last night fixing any date that appeared before 2005-11-01 since that is the first record anyone sent me. The svn has been updated. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.03 03:57:00 - [75] I have received an offer to buy the information that has been collected. I'm now asking those who contributed, what is the data worth? I'm asking this of all the people who contributed. All profits from the data will be distributed to those who contributed. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2007.02.03 22:02:00 - [76] Edited by: Menkaure on 03/02/2007 21:59:10 500k/1m per month per region So for 500k, 20 Empire regions, 12 months (encompassing all of 2006) = 120m or 1 Region, 12 months = 6m etc.
Dalknor Ithilistar	Posted - 2007.02.04 20:45:00 - [77] See that you were looking for someone to do Tash still. Not sure if you still need it, but I've sent data collected today.
Dr Slurm General Commodities	Posted - 2007.02.05 13:28:00 - [78] Maybe I forgot to update stuff but someone has completed Tash-Murkon. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.15 12:07:00 - [79] I'm finishing up the data tonight. I'm sorry I can't guarantee the accuracy of it but I place it at somewhere near 6% error ratio +/- 3%. I'm finding it almost impossible to make all the corrections that are needed. I will continue to make corrections and enter the data into the SVN. I will be emailing the data in a zip containing several formats. Sorry about the long wait, and I hope everyone is happy with the data. Check out the Market History OCR Project here
Jessamine Freedom United	Posted - 2007.02.15 17:37:00 - [80] I'm a dork with too much money, you still looking to sell that data? Might need instructions on how to use it though Originally by: Marcus TheMartin Don't call me a griefer just becase you fly a space pinata and I choose to be the one to swing the stick

Ray McCormack BIG	Posted - 2007.02.15 20:06:00 - [81] What are the most common errors you're finding, Dr Slurm? Also, send me an evemail when you've finalised details. \| How To Afford A Tech II BPO \| the all mighty BIG industrial corp from hell with a slong the size of a walrus.
Dr Slurm General Commodities	Posted - 2007.02.16 01:51:00 - [82] Originally by: Jessamine I'm a dork with too much money, you still looking to sell that data? Might need instructions on how to use it though Yes the plans to sell the data still exist. All updates will be included if you buy the data also. Originally by: Ray McCormack What are the most common errors you're finding, Dr Slurm? Also, send me an evemail when you've finalised details. I basically do this query on my database: "SELECT COUNT(*) FROM market_data WHERE avg<low OR low>high OR high<low OR avg>high GROUP BY regionName, typeName" This basically checks to make sure that the average price is somwhere in between the high and low prices. Anything outside the range is obviously a mistake. If you look above on this page you'll see the report of errors that I posted. At that time it was 5% for all the data collected. With some regions having a lot more errors then others. My problem is it basically takes me an hour to correct 100 errors and as of last count there were about 1700 errors. If other people are willing they can easily open the images they sent me and help with the error checking, but I just can't manage it on my own. What can I say I have a life. I have made several corrections by hand, but not too much of a dent. A half assed thing I could do is just average the high and low prices and replace the average just so the data doesn't look bad. Basically the most common errors are the OCR software having trouble telling the difference between 0's, 6's, 8's, and 9's because the EVE font is so similar in those numbers. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.16 04:47:00 - [83] Below is a license I wrote for this data. I would like feedback from the members about how they feel about this. Personally I have a distaste for User License Agreements. A very strong distaste. I view them more as a suggestion then anything. Quote: Eve Data Cooperative License Agreement For Cooperative Members: Definition: Data: Raw information distributed by Eve Data Cooperative consisting of statistics. Analysis: Any interpretation of the above mentioned data. Member: a person who has contributed screenshots for One year for all commodities (minerals) in a Priority* region to the Cooperative. Membership in the Eve Data Cooperative is limited to the one character that was used to contribute. This membership will last the life time of Eve-Online. Abusive members determined by a majority corp vote may possibly be dismissed and access to Data revoked. *Priority region: A priority region is any region included in this list, all other regions are considered Non-Priority: (Aridia, Derelik, Devoid, Domain, Essence, Everyshore, Genesis, Heimatar, Kador, Khanid, Kor-Azor, Lonetrek, Metropolis, Molden Heath, Sinq Laison, Tash-Murkon, The Bleak Lands, The Citadel, The Forge, Verge Vendor) A Co-operative member may do whatever they wish with this data. They may redistribute it if they feel so inclined. The only request of the cooperative is that any profits directly relating to the sale of the data (NOTE: NOT analysis) will be transfered to the Eve Data Cooperative Corporation. These profits will then be paid out in full to the entire Cooperative. This is the spirit of a cooperative. We all contributed and we shall all reap the benefits. Any analysis that members do generate from this data will be gladly hosted at http://edc.eve-commodities.com and available to all Coop members. If this analysis is to be sold is the decision solely of the author of the analysis. Profit from analysis of data hosted by Eve Data Cooperative will be redistributed as dividends to the Coop. For Non-Members: Definition: Non-Member, any individual who buys the data or analysis from Eve Data Cooperative or does not fall in the above definition of "Member". A non-member is limited in that they may not distribute this data outside of their Corporation. The Eve Data Cooperative recognizes this is not enforceable, but if people would like to enjoy this service in the future it needs to be worth it to the contributors. In this we please ask that our customers respect this license. A non-member may freely distribute any analysis they produce from the Eve Data Cooperative data for any price they feel so inclined. Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 12:43:00 - [84] Not to be critical of you slurm, but what happens when (its highly likely as ccp does this regularly to prevent macro readers) ccp changes the font, any learning algorithm used with the ocr and the data it has gleamed so far in better interpreting images is rendered useless? Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/
Block Ukx	Posted - 2007.02.16 13:06:00 - [85] Nice Work! License seems reasonable. Regarding price, you are probably in a better position to decided price since you know how much work it took to pull something like this togehter. 1,000,000 per month per region is a very good price; almost a bargain.
Menkaure Amarr Vanitas Corp.	Posted - 2007.02.16 14:16:00 - [86] Edited by: Menkaure on 16/02/2007 14:16:19 Like the license agreement, just playing with the data now. As a teaser for anyone considering buying the data, what I just pulled from it: There were 3,314,413,541,690 units of tritanium purchased in the core regions in 2006, at an average price of 1.83989979 isk per unit. That means that 6,098,188,778,938 isk was spent over the market for trit in a single year. Now, go buy the data. Slurm did you decide on a pricing plan yet? (+ excellent job btw, this'll keep me occupied for hours) EDIT: To any dev's reading this; there has been alot of speculation on how the table works and its fairly freqently shown to be a bit iffy, can we get some clarification on its exact mechanics given the scope of this project?
Dr Slurm General Commodities	Posted - 2007.02.16 17:57:00 - [87] Edited by: Dr Slurm on 16/02/2007 17:56:05 Originally by: ToxicFire Not to be critical of you slurm, but what happens when (its highly likely as ccp does this regularly to prevent macro readers) ccp changes the font, any learning algorithm used with the ocr and the data it has gleamed so far in better interpreting images is rendered useless? Not that its any of your concern, but the reader only takes about 30mins to teach to the point where it can't get any better. Originally by: Block Ukx Nice Work! License seems reasonable. Regarding price, you are probably in a better position to decided price since you know how much work it took to pull something like this togehter. 1,000,000 per month per region is a very good price; almost a bargain. I think that's a little too cheap honestly. 1mil per month per mineral would be more reasonable. Originally by: Menkaure Like the license agreement, just playing with the data now. As a teaser for anyone considering buying the data, what I just pulled from it: There were 3,314,413,541,690 units of tritanium purchased in the core regions in 2006, at an average price of 1.83989979 isk per unit. That means that 6,098,188,778,938 isk was spent over the market for trit in a single year. Now, go buy the data. Slurm did you decide on a pricing plan yet? (+ excellent job btw, this'll keep me occupied for hours) EDIT: To any dev's reading this; there has been alot of speculation on how the table works and its fairly freqently shown to be a bit iffy, can we get some clarification on its exact mechanics given the scope of this project? I think I've decided on a price plan. I want to price it to sell otherwise its unfair to the Coop members. There was a lot of work on my part for this project and I'm ok with that. It honestly would have been a hell of a lot more work if people hadn't contributed the screenshots and I did it all by myself. I'm also greatly benefiting from the data so I don't mind so much. All data is sold in one month portions. A one month portion includes all commodities(minerals). Priority(Core) Regions: 6,500,000 per month Non-Priority(Non-Core) Regions: 10,000,000 per month (subject to availability of data, if screenshots are supplied this won't be a problem. A special rate may be negotiated to obtain specifc market information from 0.0 and low sec regions. This will be dealt with on a case by case basis and is not subject to public pricing.) A little more of a teaser below. It is probably slightly more accurate then Menkaure's estimate. //Edit: I multiplied each individual row then added it to the sum to get the below figures. Estimated Amount of ISK traded in Tritanium for the year of '06 Min: 6,619,326,642,007 ISK =SUM(VOLUMEMIN) --doubtful Max: 2,699,438,616,943,265 ISK =SUM(VOLUMEMAX) --extremely unlikely Avg: 7,052,425,308,218 ISK =SUM(VOLUME*AVG) --probably about right Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 21:17:00 - [88] p.s. the links in your original post don't appear to be working Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/
Dr Slurm General Commodities	Posted - 2007.02.16 22:30:00 - [89] Originally by: ToxicFire p.s. the links in your original post don't appear to be working My webhost is currently having issues. They should be back today I'm imagining, but there isn't anything too important on the front page. Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 22:56:00 - [90] Originally by: Dr Slurm Originally by: ToxicFire p.s. the links in your original post don't appear to be working My webhost is currently having issues. They should be back today I'm imagining, but there isn't anything too important on the front page. Gud gud, still looking for any more investors aside from the regions you havent got data for yet, im actually quite interested in contributing Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/

Dr Slurm General Commodities	Posted - 2007.02.16 23:09:00 - [91] Edited by: Dr Slurm on 16/02/2007 23:06:05 If you'd like to sign up for the Non-Priority regions you are welcome too. Unfortunately you can't buy a stake in [EDC] with ISK, only contribute data. You are welcome to purchase data also.
Zugor Ikatin Corporation 35	Posted - 2007.03.19 00:11:00 - [92] Is this project still alive? It would be nice to see some data on Zydrine. ------------------------------------------------------------ "We that are strong ought to bear the infirmities of the weak but not to please ourselves." - Romans 15:1
Dr Slurm General Commodities	Posted - 2007.03.19 15:47:00 - [93] The project is no longer actively being worked on. I of course still have all the data. I have distributed it to the Coop members. If you would like to purchase the zydrine data you can contact me ingame.
Dr Slurm General Commodities	Posted - 2007.04.19 01:18:00 - [94] I've decided to change the pricing scheme. All collected data can be purchased for 200m ISK.
Galgorth	Posted - 2007.09.29 04:16:00 - [95] If you could do this again with up-to-date data, I would purchase it for 200 mil.
Dr Slurm General Commodities	Posted - 2007.09.29 04:28:00 - [96] Originally by: Galgorth If you could do this again with up-to-date data, I would purchase it for 200 mil. I might conduct this again, but I certainly won't sell you the data. <sig> Tired of the inane ramblings of the incompetent? Click here </sig>
Galgorth	Posted - 2007.09.29 04:30:00 - [97] Originally by: Dr Slurm Originally by: Galgorth If you could do this again with up-to-date data, I would purchase it for 200 mil. I might conduct this again, but I certainly won't sell you the data. Then I'll have to steal it from you
Dr Slurm General Commodities	Posted - 2007.09.29 05:21:00 - [98] Originally by: Galgorth Originally by: Dr Slurm Originally by: Galgorth If you could do this again with up-to-date data, I would purchase it for 200 mil. I might conduct this again, but I certainly won't sell you the data. Then I'll have to steal it from you You can think that. <sig> Tired of the inane ramblings of the incompetent? Click here </sig>


Pages: 1 2 3 4 :: [one page]

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.