EVE Search - Importing EVE market cache data to excel

Check My IP Information

All Channels

EVE Technology Lab

Importing EVE market cache data to excel

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: [1] :: one page

Author	Thread Statistics \| Show CCP posts - 1 post(s)
Casshern Price Ministry of War Amarr Empire 0	Posted - 2015.01.23 14:52:58 - [1] - Quote Hello forums o/ I was wondering if anyone out there has the know-how on importing the EVE market cache data you get every time you fetch item data to an Excel 2013 file. I know that some programs like EVE-Mentat can do it, but the program doesn't have the layout I desire. I've seen a lot of tutorials on how to do this with Google Sheets, but the data isn't as accurate as I'd like it to be. I'll say again please do not link a tutorial on how to do this in Google Sheets, I've already made 2 and they don't have the performance that I'm seeking. Basically what I'm looking to do is make a spreadsheet that will hold buy/sell/quantity/etc. data for multiple stations. For instance, if I had a station trader in Jita and another character in any other system, I'd like to be able to fetch certain item data in-game on both characters and then import the data to the Excel file. Thanks in advance for any help, Price
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4761	Posted - 2015.01.23 15:09:29 - [2] - Quote https://www.fuzzwork.co.uk/2013/06/22/importing-price-data-into-spreadsheets/ doesn't load from the cache though. It's pulling from eve-central. Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Hel O'Ween Men On A Mission 105	Posted - 2015.01.24 16:14:37 - [3] - Quote I haven't done it yet, as that's a task which requires either some from-scratch programming (either creating a native Win32 or COM DLL consumable by Windows applications, which either scraps the cache itself or might act as an EMDR endpoint) or cobble together a tool chain by using i.e. Entity's reference Phyton stuff to scrap the cache, produce some kind of "readable" files from it and import those into Excel. Either way - it's most likely something you won't put together within a couple of lazy sunday afternoon hours. EVEWalletAware - an offline wallet manager.
Casshern Price Ministry of War Amarr Empire 0	Posted - 2015.01.24 23:37:10 - [4] - Quote Steve Ronuken wrote: https://www.fuzzwork.co.uk/2013/06/22/importing-price-data-into-spreadsheets/ doesn't load from the cache though. It's pulling from eve-central. Yeah, that's exactly what I don't want it to do. Instead of importing from an online source, I would prefer to import the data from my own cache, that way I know it's completely reliable. Thanks though. Hel O'Ween wrote: I haven't done it yet, as that's a task which requires either some from-scratch programming (either creating a native Win32 or COM DLL consumable by Windows applications, which either scraps the cache itself or might act as an EMDR endpoint) or cobble together a tool chain by using i.e. Entity's reference Phyton stuff to scrap the cache, produce some kind of "readable" files from it and import those into Excel. Either way - it's most likely something you won't put together within a couple of lazy sunday afternoon hours. Darn, that's definitely discouraging to read. I took a highschool class on programming and we covered some Python, but definitely not enough for me to create a program from scratch :P
Hel O'Ween Men On A Mission 105	Posted - 2015.01.25 10:54:43 - [5] - Quote Forgot to mention the new CREST APIs. Not sure though if someone has put something easy to use together yet. It's also worth mentioning that for the time being, the nowadays practiced cache scraping - although technically a vaiolation of the EULA - is tolerated (but discouraged) by CCP. But CCP FoxFour (I think it was him) has mentioned that with the release of the CREST market API, the EULA on cache scraping will be enforced rather sooner than later. EVEWalletAware - an offline wallet manager.
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.01.28 06:24:19 - [6] - Quote Ooo, Intriguing. FoxFour mentions in the blog post that this information is real-time, does this mean that soon all those spreadsheets that import data from places like EVE-Central will now be obsolete, or do I just have no idea how to read the code?
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4807	Posted - 2015.01.28 11:36:27 - [7] - Quote Casshern Price wrote: Ooo, Intriguing. FoxFour mentions in the blog post that this information is real-time, does this mean that soon all those spreadsheets that import data from places like EVE-Central will now be obsolete, or do I just have no idea how to read the code? tbh, most people will continue to use eve-central. The feed is useful, but it's substantially more work than asking eve-central for the data. And when you only need approximate pricing, it's more work than you need. Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.01.28 23:35:43 - [8] - Quote Steve Ronuken wrote: Casshern Price wrote: Ooo, Intriguing. FoxFour mentions in the blog post that this information is real-time, does this mean that soon all those spreadsheets that import data from places like EVE-Central will now be obsolete, or do I just have no idea how to read the code? tbh, most people will continue to use eve-central. The feed is useful, but it's substantially more work than asking eve-central for the data. And when you only need approximate pricing, it's more work than you need. Ah, I see. I'm trying to stay away from those approximate prices though, haha. When you're trading, you kind of need prices and volume data to be as fresh as possible; I'm sure you know that though :P Do you know if it's at all possible to import that kind of data with this CREST API or am I just out of luck in general in this department?
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4819	Posted - 2015.01.29 02:10:04 - [9] - Quote I'd say trying to do it in Excel is going to be painful. Visual basic script has really poor json handling. However, what might be possible, is an application which exists purely to pull market data and store a file locally, which Excel could then interact with. Probably putting it into a database would be the best option, with an odbc connection for Excel. It's not the fastest thing to update, when I was doing it in php, it was taking around half a second per request. Does that sound like the kind of thing you'd want? What kind of information would you be looking for? Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.01.29 07:10:35 - [10] - Quote Well, just accurate buy/sell order prices, volume, etc. I thought previously that I could do this by taking the data from my cache, but you said that method isn't exactly condoned by CCP. You mentioned that FoxFour post though and I saw real-time and I got all excited, haha. Would using those APIs in Google Sheets be easier than Excel? I don't have a problem with the sheets themselves, just the source of the data, which isn't all that reliable. Can the APIs even relay the data I'm looking for, or is it something else entirely? Basically what I'm looking to do is manually scan (for now) through items in two systems (for now) and upload the data into some kind of Excel/Google Sheets program so that I can work the math and find some ever delightful profits. Sorry if I'm coming off a bit confusing, I'm new to all this API and such stuff.


CCP FoxFour C C P C C P Alliance 3914	Posted - 2015.01.29 09:48:30 - [11] - Quote Casshern Price wrote: Well, just accurate buy/sell order prices, volume, etc. I thought previously that I could do this by taking the data from my cache, but you said that method isn't exactly condoned by CCP. You mentioned that FoxFour post though and I saw real-time and I got all excited, haha. Would using those APIs in Google Sheets be easier than Excel? I don't have a problem with the sheets themselves, just the source of the data, which isn't all that reliable. Can the APIs even relay the data I'm looking for, or is it something else entirely? Basically what I'm looking to do is manually scan (for now) through items in two systems (for now) and upload the data into some kind of Excel/Google Sheets program so that I can work the math and find some ever delightful profits. Sorry if I'm coming off a bit confusing, I'm new to all this API and such stuff. I would recommend using the EVE Central API. It is a lot easier to work with, especially for spreadsheets, and they are pulling their data from CREST so should pretty much be real time. Their API if I remember should also have all the information you are after. @CCP_FoxFour // Technical Designer // Team Size Matters Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Hel O'Ween Men On A Mission 105	Posted - 2015.01.29 17:35:52 - [12] - Quote Steve Ronuken wrote: Visual basic script has really poor json handling. Poor? Non-existent is closer to the truth. I once went "shopping" for some VBA JSON library/code and the only decent one I was able to find was this one: http://www.codeproject.com/Articles/720368/VB-JSON-Parser-Improved-Performance. Did a few simple test, which went OK, but haven't really used it in production. EVEWalletAware - an offline wallet manager.
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.01.30 00:32:16 - [13] - Quote CCP FoxFour wrote: Casshern Price wrote: Well, just accurate buy/sell order prices, volume, etc. I thought previously that I could do this by taking the data from my cache, but you said that method isn't exactly condoned by CCP. You mentioned that FoxFour post though and I saw real-time and I got all excited, haha. Would using those APIs in Google Sheets be easier than Excel? I don't have a problem with the sheets themselves, just the source of the data, which isn't all that reliable. Can the APIs even relay the data I'm looking for, or is it something else entirely? Basically what I'm looking to do is manually scan (for now) through items in two systems (for now) and upload the data into some kind of Excel/Google Sheets program so that I can work the math and find some ever delightful profits. Sorry if I'm coming off a bit confusing, I'm new to all this API and such stuff. I would recommend using the EVE Central API. It is a lot easier to work with, especially for spreadsheets, and they are pulling their data from CREST so should pretty much be real time. Their API if I remember should also have all the information you are after. Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs?
Hel O'Ween Men On A Mission 105	Posted - 2015.01.30 18:43:41 - [14] - Quote Casshern Price wrote: Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs? The EVE Central devs stated elsewhere i this forum that they are about to implement the CREST market data API, so be a bit patient with them. Currently there's no marketdata API, so they depend on players uploading data to them. So, if you feel their data is outdated, you haven't used their uploader (or EVEMon) to update the outdated data. EVEWalletAware - an offline wallet manager.
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.02.01 01:02:05 - [15] - Quote Hel O'Ween wrote: Casshern Price wrote: Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs? The EVE Central devs stated elsewhere i this forum that they are about to implement the CREST market data API, so be a bit patient with them. Currently there's no marketdata API, so they depend on players uploading data to them. So, if you feel their data is outdated, you haven't used their uploader (or EVEMon) to update the outdated data. Great news I'm glad to hear that that fact that the data is inaccurate is because they're not actually pulling from the API yet. Yeah, I thought that the players were using some kind of uploader for the data, I saw that they do that on 'eve-marketdata.com' as well. I would never use the uploader myself haha, while it would give me up-to-date data, it would also send the data to every other person that has a spreadsheet, effectively turning it into a double-edged sword. If I find a deal, I'd like as few people to know about it as manageable, that way I can squeeze every lost drop of ISK I can out of it.
SJ Astralana Syncore 89	Posted - 2015.02.01 03:35:57 - [16] - Quote Casshern Price wrote: Hel O'Ween wrote: Currently there's no marketdata API Great news I'm glad to hear that that fact that the data is inaccurate is because they're not actually pulling from the API yet. I'm not sure what the previous comment is trying to say, but market history and current orders certainly are available via CREST and I'm using it to mine profits from current data. The accuracy of information compared to existing 3rd party services is refreshing to say the least. It's also a certainty that the aggregators will transition away from uploaded data, which means hidden profits will be a thing of the past. Hyperdrive your production business: Eve Production Manager
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.02.01 03:54:26 - [17] - Quote SJ Astralana wrote: Casshern Price wrote: Hel O'Ween wrote: Currently there's no marketdata API Great news I'm glad to hear that that fact that the data is inaccurate is because they're not actually pulling from the API yet. I'm not sure what the previous comment is trying to say, but market history and current orders certainly are available via CREST and I'm using it to mine profits from current data. The accuracy of information compared to existing 3rd party services is refreshing to say the least. It's also a certainty that the aggregators will transition away from uploaded data, which means hidden profits will be a thing of the past. Would you be willing to share with me how you're pulling the data? I can't for the life of me find anything helpful.
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4859	Posted - 2015.02.01 05:09:10 - [18] - Quote Hel O'Ween wrote: Casshern Price wrote: Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs? The EVE Central devs stated elsewhere i this forum that they are about to implement the CREST market data API, so be a bit patient with them. Currently there's no marketdata API, so they depend on players uploading data to them. So, if you feel their data is outdated, you haven't used their uploader (or EVEMon) to update the outdated data. Umm, there is an API, which is up and running. It's just not the easiest of things to use, in anything but a full blown language. Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
SJ Astralana Syncore 89	Posted - 2015.02.01 05:55:02 - [19] - Quote Casshern Price wrote: Would you be willing to share with me how you're pulling the data? I can't for the life of me find anything helpful. The endpoints are https://crest-tq.eveonline.com/market/{regionId}/orders/{buy \| sell}/?type=https://api.eveonline.com/types/{typeId}/ and https://crest-tq.eveonline.com/market/{regionId}/types/{typeId}/history/, but as pointed out this is hard stuff if you're not up to commercial web development skills. There are APIs available to abstract some of this away, but consuming an API or rolling your own is a tiny fraction of the effort of wrapping your head around OAuth. A short list of your hurdles are redirecting to EVE and receiving the response on your web service; constructing web requests for both auth and data with the necessary headers; managing access and refresh tokens; etc. CCP certainly didn't make this easy for the casual consumer, who will be far better off waiting for a 3rd party to do the legwork. Hyperdrive your production business: Eve Production Manager
Casshern Price Controlled Chemical Exposure Ltd. 0	Posted - 2015.02.01 06:34:09 - [20] - Quote SJ Astralana wrote: The endpoints are https://crest-tq.eveonline.com/market/{regionId}/orders/{buy \| sell}/?type=https://api.eveonline.com/types/{typeId}/ and https://crest-tq.eveonline.com/market/{regionId}/types/{typeId}/history/, but as pointed out this is hard stuff if you're not up to commercial web development skills. There are APIs available to abstract some of this away, but consuming an API or rolling your own is a tiny fraction of the effort of wrapping your head around OAuth. A short list of your hurdles are redirecting to EVE and receiving the response on your web service; constructing web requests for both auth and data with the necessary headers; managing access and refresh tokens; etc. CCP certainly didn't make this easy for the casual consumer, who will be far better off waiting for a 3rd party to do the legwork. Yeah, I'll have to wait if that's what it takes, because most of what you said went right over my head.

Hel O'Ween Men On A Mission 105	Posted - 2015.02.03 17:55:43 - [21] - Quote Steve Ronuken wrote: Hel O'Ween wrote: Casshern Price wrote: Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs? The EVE Central devs stated elsewhere i this forum that they are about to implement the CREST market data API, so be a bit patient with them. Currently there's no marketdata API, so they depend on players uploading data to them. So, if you feel their data is outdated, you haven't used their uploader (or EVEMon) to update the outdated data. Umm, there is an API, which is up and running. It's just not the easiest of things to use, in anything but a full blown language. There's an (CCP) XML API for market data? EVEWalletAware - an offline wallet manager.
Casshern Price Controlled Chemical Exposure Ltd. 1	Posted - 2015.02.05 09:17:43 - [22] - Quote Is there?? Isn't there?!?! The anticipation is killing me!! Haha
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4875	Posted - 2015.02.05 11:10:01 - [23] - Quote Hel O'Ween wrote: Steve Ronuken wrote: Hel O'Ween wrote: Casshern Price wrote: Ah, does their API cover that? I had no idea :P I've noticed though that their order prices will sometimes differ drastically from the actual orders, leading me to believe that the data was pretty old. Is this just the current limitations of the 'real-time' APIs? The EVE Central devs stated elsewhere i this forum that they are about to implement the CREST market data API, so be a bit patient with them. Currently there's no marketdata API, so they depend on players uploading data to them. So, if you feel their data is outdated, you haven't used their uploader (or EVEMon) to update the outdated data. Umm, there is an API, which is up and running. It's just not the easiest of things to use, in anything but a full blown language. There's an (CCP) XML API for market data? (could have sworn I'd posted a response) Sorry miscommunication on the type of API. Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Jess Ohaya Science and Trade Institute Caldari State 0	Posted - 2015.02.06 07:31:33 - [24] - Quote Hi All, Sorry to be the daft one - we've talked about it, and clarified it a couple of times - but I still don't understand what the answer actually is... Presuming: (1) I know just enough about how the internets work to have myself a local database, and via mostly someone else's PHP I am able to populate said database with current market prices for a large number of items in a large number of systems by means of retrieving and storing prices from the Eve-Central API (2) I don't expect I will ever understand how CREST really works enough for me to rebuild my current setup such that instead of asking Eve-Centrals api for current market data, I could instead ask crest. (3) Outside of Jita prices, in order to get 'current' market data I need to cycle through my items in-game with Contribtastic open such that I actually get 'current' market prices from eve-central Clarifying Answers please: (A) Will it be possible for me to wait for Eve-Central to finish whatever it is that they are working on re: Crest such that my application will still work exactly how it does now ? (B) Contribtastic currently has a note on it saying that "they are moving to crest soon, obviating the need for contribtastic" - Once contribtastic goes away, will Eve-Centrals APIs still work as they do now ? or is the work that eve-central is doing similar to what we have recently seen with other web-crest-market apps that just let you browse the market in a web browser ? (C) Is it likely that someone will build a better(?) way for those of us who need to download and store a lot of market data to then work on locally/offline ? It looks like I should actually be asking someone at Eve-Central... but hopefully someone here can just give a definitive Yes/No to A, B and C - Thanks !
Steve Ronuken Fuzzwork Enterprises Vote Steve Ronuken for CSM 4879	Posted - 2015.02.06 15:38:29 - [25] - Quote A: Yes B: The last I heard, they're just feeding eve central from CREST. No perceptible change from users. C: Yes. https://github.com/fuzzysteve/CrestLibrary does bits. Woo! CSM 9! Fuzzwork Enterprises Twitter: @fuzzysteve on Twitter
Jess Ohaya Science and Trade Institute Caldari State 0	Posted - 2015.02.06 22:38:04 - [26] - Quote Awesome - Thanks for clearing that up Steve!
Def Monk Phoenix Navy Research and Development Phoenix Naval Systems 11	Posted - 2015.02.08 20:26:41 - [27] - Quote While I'm always all for people learning how to handle APIs and develop third-party apps for Eve, I feel like a lot of what people are asking for in this thread already gets covered by applications like Evernus. It's a great program, and does anything I could have wanted for market trading (Including prices across new eden, comparisons, margins, etc. from Crest). I felt I should pop in and mention it to try and save people some time and possible headache.


Pages: [1] :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.