EVE Search - Market Data OCR Project

All Channels

Market Discussions

Market Data OCR Project

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 [3] 4 :: one page
Author	Thread Statistics \| Show CCP posts - 0 post(s)
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.20 07:00:00 - [61] If Icy One hasn't got you tash within a week or so I'll swing round and get it. Just gimme a bell ingame or on here or somesuch.
Dr Slurm General Commodities	Posted - 2007.01.22 02:00:00 - [62] Tash-Murkon is back up for grabs and the 500 shares that go with it. The next person to respond with interest gets it. (Just to be fair Menk) Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.22 13:16:00 - [63] Edited by: Dr Slurm on 22/01/2007 13:18:16 Region # Errors # Entries Ratio % Derelik 24028700.08 The Forge 6127300.02 Lonetrek 7929800.03 Heimatar 21427320.08 The Citadel 7429880.02 Devoid 12527990.04 The Bleak Lands 12428520.04 Metropolis 7929820.03 Domain 9227300.03 Khanid 10128210.04 Kador 4428410.02 Essence 22526230.09 Kor-Azor 27728180.1 Verge Vendor 9929750.03 SUM1834397410.05 This is a summary from the database that I pulled. It shows the number of errors that I know exist for sure. This was just a simple check to make sure the low, high and avg related to each other properly. More errors then this probably exist. The last line is the total. Slurm Check out the Market History OCR Project here
Feya Kortimna	Posted - 2007.01.25 05:21:00 - [64] Do you have some 'automagical' error-correction in mind? I'll be happy to poke around in the screenshots / data I provided to clean up those errors, if the solution is manual checking. If you send me a TortoiseSVN userid, I will be able to get at the error-rich data, right?
Dr Slurm General Commodities	Posted - 2007.01.25 05:51:00 - [65] Originally by: Feya Kortimna Do you have some 'automagical' error-correction in mind? I'll be happy to poke around in the screenshots / data I provided to clean up those errors, if the solution is manual checking. If you send me a TortoiseSVN userid, I will be able to get at the error-rich data, right? Well the regions with an error rate higher then 3% I was just going to do again since its not that hard. Just a little time consuming. The others I was just going to look through to see if it as something obvious. I have to say for beginning run in an OCR project %5 error ratio is pretty decent, but then again I'm sure there are errors that I couldn't easily check for. I will check in all my data and send you a login. Sidurii notified me ingame that they will be doing the Placid region and Tash-Murkon. As for an update, I'm finishing up Genesis right now, and as soon as I receive Tash I will process it. The non-priority regions will wait until the Priority regions are fully processed and published for the membership. I hope to make my self imposed deadline of the end of the month, but I worry there will be a lot errors in the data, devaluing its worth. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.01.25 07:04:00 - [66] I just checked in a .sql and a .csv dump of the data into the SVN repository. It's pretty easy to use, check it out. I'd link it but I'm tired and you all should have a received an email from opensvn.csie.org with a login and password. Enjoy! I'll be updating it as I and others correct errors. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.27 18:12:00 - [67] Edited by: Menkaure on 27/01/2007 18:10:29 Originally by: Dr Slurm I just checked in a .sql and a .csv dump of the data into the SVN repository. It's pretty easy to use, check it out. I'd link it but I'm tired and you all should have a received an email from opensvn.csie.org with a login and password. Enjoy! I'll be updating it as I and others correct errors. One small quibble, had a look at the data but can you chance the CSV so that the data is split by something shorter than "& q u o t ; ; & q u o t ;" ?? Like, just a comma or ;; or something? (heh, forum kept on eating it so put loadsa spaces in to split it up. Spaces aint present in the actual CSV. (when i put in what it actually is, it comes out as ";" ) Thats too long for openoffice calc to split it into columns (dunno about excel) and I don't particularly wanna kill my computer for a few hours by replacing it in notepad or something. (yes, if there's an easier way to do it then tell me- I am a newb at this kinda stuff)
Motivated Prophet Zerodot Schools Power Corrupts Industry's	Posted - 2007.01.27 18:28:00 - [68] I believe Wordpad (write.exe in your "Run" box) will not update the screen until after the replacement operation is complete, which means it won't take hours, just seconds. Of course, if the entire thing is HTML-encoded, just stick "<PRE>" at the start (sans quotes), save it to a file, change the file's extension to .HTM, open it in a browser, and copy and paste the data back where you need it. MP
Kua Burrow	Posted - 2007.01.27 19:57:00 - [69] Originally by: Dr Slurm The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. Um this what you're doing sounds interesting and useful but at the risk of sounding dumb I'm not really sure what it is you are actually doing . The above quote seems to be the crux of the matter - would someone expand on it and explain in layman's terms what this project involves and (more importantly) what it will leave us with that will mean we are better off than b4? Thanks.
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.27 20:04:00 - [70] Originally by: Motivated Prophet I believe Wordpad (write.exe in your "Run" box) will not update the screen until after the replacement operation is complete, which means it won't take hours, just seconds. Of course, if the entire thing is HTML-encoded, just stick "<PRE>" at the start (sans quotes), save it to a file, change the file's extension to .HTM, open it in a browser, and copy and paste the data back where you need it. MP Notepad I left for about 2 hours trying to do it before i closed it down, but you're right, wordpad is better. Managed it after about half an hour. Thanks for the tip.
Dr Slurm General Commodities	Posted - 2007.01.28 00:33:00 - [71] Wow I didn't even look at what it put out. I'll fix it and check in the changes Check out the Market History OCR Project here
gazarsgo Einherjar Rising	Posted - 2007.01.30 15:35:00 - [72] from the 1.3 notes: Quote: + Market History graphs should now display the correct values of ISK instead of cropping the most significant digit. Does this affect your work to date at all?
Menkaure Amarr Vanitas Corp.	Posted - 2007.01.30 15:58:00 - [73] Originally by: gazarsgo from the 1.3 notes: Quote: + Market History graphs should now display the correct values of ISK instead of cropping the most significant digit. Does this affect your work to date at all? No that refers to the graph, this project uses the table.
Dr Slurm General Commodities	Posted - 2007.01.31 14:01:00 - [74] I did a massive replace to the mysql database last night fixing any date that appeared before 2005-11-01 since that is the first record anyone sent me. The svn has been updated. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.03 03:57:00 - [75] I have received an offer to buy the information that has been collected. I'm now asking those who contributed, what is the data worth? I'm asking this of all the people who contributed. All profits from the data will be distributed to those who contributed. Check out the Market History OCR Project here
Menkaure Amarr Vanitas Corp.	Posted - 2007.02.03 22:02:00 - [76] Edited by: Menkaure on 03/02/2007 21:59:10 500k/1m per month per region So for 500k, 20 Empire regions, 12 months (encompassing all of 2006) = 120m or 1 Region, 12 months = 6m etc.
Dalknor Ithilistar	Posted - 2007.02.04 20:45:00 - [77] See that you were looking for someone to do Tash still. Not sure if you still need it, but I've sent data collected today.
Dr Slurm General Commodities	Posted - 2007.02.05 13:28:00 - [78] Maybe I forgot to update stuff but someone has completed Tash-Murkon. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.15 12:07:00 - [79] I'm finishing up the data tonight. I'm sorry I can't guarantee the accuracy of it but I place it at somewhere near 6% error ratio +/- 3%. I'm finding it almost impossible to make all the corrections that are needed. I will continue to make corrections and enter the data into the SVN. I will be emailing the data in a zip containing several formats. Sorry about the long wait, and I hope everyone is happy with the data. Check out the Market History OCR Project here
Jessamine Freedom United	Posted - 2007.02.15 17:37:00 - [80] I'm a dork with too much money, you still looking to sell that data? Might need instructions on how to use it though Originally by: Marcus TheMartin Don't call me a griefer just becase you fly a space pinata and I choose to be the one to swing the stick
Ray McCormack BIG	Posted - 2007.02.15 20:06:00 - [81] What are the most common errors you're finding, Dr Slurm? Also, send me an evemail when you've finalised details. \| How To Afford A Tech II BPO \| the all mighty BIG industrial corp from hell with a slong the size of a walrus.
Dr Slurm General Commodities	Posted - 2007.02.16 01:51:00 - [82] Originally by: Jessamine I'm a dork with too much money, you still looking to sell that data? Might need instructions on how to use it though Yes the plans to sell the data still exist. All updates will be included if you buy the data also. Originally by: Ray McCormack What are the most common errors you're finding, Dr Slurm? Also, send me an evemail when you've finalised details. I basically do this query on my database: "SELECT COUNT(*) FROM market_data WHERE avg<low OR low>high OR high<low OR avg>high GROUP BY regionName, typeName" This basically checks to make sure that the average price is somwhere in between the high and low prices. Anything outside the range is obviously a mistake. If you look above on this page you'll see the report of errors that I posted. At that time it was 5% for all the data collected. With some regions having a lot more errors then others. My problem is it basically takes me an hour to correct 100 errors and as of last count there were about 1700 errors. If other people are willing they can easily open the images they sent me and help with the error checking, but I just can't manage it on my own. What can I say I have a life. I have made several corrections by hand, but not too much of a dent. A half assed thing I could do is just average the high and low prices and replace the average just so the data doesn't look bad. Basically the most common errors are the OCR software having trouble telling the difference between 0's, 6's, 8's, and 9's because the EVE font is so similar in those numbers. Check out the Market History OCR Project here
Dr Slurm General Commodities	Posted - 2007.02.16 04:47:00 - [83] Below is a license I wrote for this data. I would like feedback from the members about how they feel about this. Personally I have a distaste for User License Agreements. A very strong distaste. I view them more as a suggestion then anything. Quote: Eve Data Cooperative License Agreement For Cooperative Members: Definition: Data: Raw information distributed by Eve Data Cooperative consisting of statistics. Analysis: Any interpretation of the above mentioned data. Member: a person who has contributed screenshots for One year for all commodities (minerals) in a Priority* region to the Cooperative. Membership in the Eve Data Cooperative is limited to the one character that was used to contribute. This membership will last the life time of Eve-Online. Abusive members determined by a majority corp vote may possibly be dismissed and access to Data revoked. *Priority region: A priority region is any region included in this list, all other regions are considered Non-Priority: (Aridia, Derelik, Devoid, Domain, Essence, Everyshore, Genesis, Heimatar, Kador, Khanid, Kor-Azor, Lonetrek, Metropolis, Molden Heath, Sinq Laison, Tash-Murkon, The Bleak Lands, The Citadel, The Forge, Verge Vendor) A Co-operative member may do whatever they wish with this data. They may redistribute it if they feel so inclined. The only request of the cooperative is that any profits directly relating to the sale of the data (NOTE: NOT analysis) will be transfered to the Eve Data Cooperative Corporation. These profits will then be paid out in full to the entire Cooperative. This is the spirit of a cooperative. We all contributed and we shall all reap the benefits. Any analysis that members do generate from this data will be gladly hosted at http://edc.eve-commodities.com and available to all Coop members. If this analysis is to be sold is the decision solely of the author of the analysis. Profit from analysis of data hosted by Eve Data Cooperative will be redistributed as dividends to the Coop. For Non-Members: Definition: Non-Member, any individual who buys the data or analysis from Eve Data Cooperative or does not fall in the above definition of "Member". A non-member is limited in that they may not distribute this data outside of their Corporation. The Eve Data Cooperative recognizes this is not enforceable, but if people would like to enjoy this service in the future it needs to be worth it to the contributors. In this we please ask that our customers respect this license. A non-member may freely distribute any analysis they produce from the Eve Data Cooperative data for any price they feel so inclined. Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 12:43:00 - [84] Not to be critical of you slurm, but what happens when (its highly likely as ccp does this regularly to prevent macro readers) ccp changes the font, any learning algorithm used with the ocr and the data it has gleamed so far in better interpreting images is rendered useless? Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/
Block Ukx	Posted - 2007.02.16 13:06:00 - [85] Nice Work! License seems reasonable. Regarding price, you are probably in a better position to decided price since you know how much work it took to pull something like this togehter. 1,000,000 per month per region is a very good price; almost a bargain.
Menkaure Amarr Vanitas Corp.	Posted - 2007.02.16 14:16:00 - [86] Edited by: Menkaure on 16/02/2007 14:16:19 Like the license agreement, just playing with the data now. As a teaser for anyone considering buying the data, what I just pulled from it: There were 3,314,413,541,690 units of tritanium purchased in the core regions in 2006, at an average price of 1.83989979 isk per unit. That means that 6,098,188,778,938 isk was spent over the market for trit in a single year. Now, go buy the data. Slurm did you decide on a pricing plan yet? (+ excellent job btw, this'll keep me occupied for hours) EDIT: To any dev's reading this; there has been alot of speculation on how the table works and its fairly freqently shown to be a bit iffy, can we get some clarification on its exact mechanics given the scope of this project?
Dr Slurm General Commodities	Posted - 2007.02.16 17:57:00 - [87] Edited by: Dr Slurm on 16/02/2007 17:56:05 Originally by: ToxicFire Not to be critical of you slurm, but what happens when (its highly likely as ccp does this regularly to prevent macro readers) ccp changes the font, any learning algorithm used with the ocr and the data it has gleamed so far in better interpreting images is rendered useless? Not that its any of your concern, but the reader only takes about 30mins to teach to the point where it can't get any better. Originally by: Block Ukx Nice Work! License seems reasonable. Regarding price, you are probably in a better position to decided price since you know how much work it took to pull something like this togehter. 1,000,000 per month per region is a very good price; almost a bargain. I think that's a little too cheap honestly. 1mil per month per mineral would be more reasonable. Originally by: Menkaure Like the license agreement, just playing with the data now. As a teaser for anyone considering buying the data, what I just pulled from it: There were 3,314,413,541,690 units of tritanium purchased in the core regions in 2006, at an average price of 1.83989979 isk per unit. That means that 6,098,188,778,938 isk was spent over the market for trit in a single year. Now, go buy the data. Slurm did you decide on a pricing plan yet? (+ excellent job btw, this'll keep me occupied for hours) EDIT: To any dev's reading this; there has been alot of speculation on how the table works and its fairly freqently shown to be a bit iffy, can we get some clarification on its exact mechanics given the scope of this project? I think I've decided on a price plan. I want to price it to sell otherwise its unfair to the Coop members. There was a lot of work on my part for this project and I'm ok with that. It honestly would have been a hell of a lot more work if people hadn't contributed the screenshots and I did it all by myself. I'm also greatly benefiting from the data so I don't mind so much. All data is sold in one month portions. A one month portion includes all commodities(minerals). Priority(Core) Regions: 6,500,000 per month Non-Priority(Non-Core) Regions: 10,000,000 per month (subject to availability of data, if screenshots are supplied this won't be a problem. A special rate may be negotiated to obtain specifc market information from 0.0 and low sec regions. This will be dealt with on a case by case basis and is not subject to public pricing.) A little more of a teaser below. It is probably slightly more accurate then Menkaure's estimate. //Edit: I multiplied each individual row then added it to the sum to get the below figures. Estimated Amount of ISK traded in Tritanium for the year of '06 Min: 6,619,326,642,007 ISK =SUM(VOLUMEMIN) --doubtful Max: 2,699,438,616,943,265 ISK =SUM(VOLUMEMAX) --extremely unlikely Avg: 7,052,425,308,218 ISK =SUM(VOLUME*AVG) --probably about right Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 21:17:00 - [88] p.s. the links in your original post don't appear to be working Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/
Dr Slurm General Commodities	Posted - 2007.02.16 22:30:00 - [89] Originally by: ToxicFire p.s. the links in your original post don't appear to be working My webhost is currently having issues. They should be back today I'm imagining, but there isn't anything too important on the front page. Check out the Market History OCR Project here
ToxicFire Phoenix Knights	Posted - 2007.02.16 22:56:00 - [90] Originally by: Dr Slurm Originally by: ToxicFire p.s. the links in your original post don't appear to be working My webhost is currently having issues. They should be back today I'm imagining, but there isn't anything too important on the front page. Gud gud, still looking for any more investors aside from the regions you havent got data for yet, im actually quite interested in contributing Join the save Stargate SG1 Campaign Today! http://savestargatesg1.com/

Pages: 1 2 [3] 4 :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.