Pages: 1 2 3 4 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.23 00:02:00 -
[1]
Greetings,
The last few days I've been working on an extension of eve-marketdata, in the spirit of eve-metrics.
I'm too tired to really write much about it, but I guess it all speaks for itself. It's a marketdata lookup tool, allows you to upload data with an uploader (sends it to eve-marketdata.com too). And if you're lucky, and someone is using the upload webpage to upload data in a region that you look up the prices for, you can retrieve live pricing data.
Take a look at: http://www.evemarketeer.com/
If you've got questions and or comments, I'd be very happy to hear them. I'll work towards incorporating any good suggestions into the site.
Callean --- "A fool flatters himself, a wise man flatters the fool."
Co-founder of NEMIA. |
DominusBellus
|
Posted - 2011.07.23 09:23:00 -
[2]
To get a bit more enthusiasm for your project, what benefits will your site be offering over and above what eve-marketdata currently provides? I assume you'll have more tools than just the lookup market prices that eve-marketdata gives?
Also, are you pulling data from eve-marketdata's api, or solely relying on your own uploader for market information? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.23 09:32:00 -
[3]
Edited by: Callean Drevus on 23/07/2011 09:33:46 Yes, there will be many things that I'm hoping to add that are currently not in eve-marketdata. There are already a few things which you can lookup on evemarketeer that are not visible on evemarketdata (including a cost estimate of an item were you to build it from scratch).
And yes, as a matter of fact, I pull the data from eve-marketdata every day at the moment. All updates done with my uploader as sent to both eve-marketdata and evemarketeer. --- "A fool flatters himself, a wise man flatters the fool."
Co-founder of NEMIA. |
Max Kolonko
Caldari Worm Nation Ash Alliance
|
Posted - 2011.07.23 10:27:00 -
[4]
I like that you send data also to eve-marketdata Max Kolonko |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.23 11:06:00 -
[5]
Edited by: Callean Drevus on 23/07/2011 11:17:34
Originally by: Max Kolonko I like that you send data also to eve-marketdata
I'm using his data, it's only fair I return the favour.
I tried sending the data to eve-central too, but their upload tool/format is just too convoluted.
EDIT: One thing I will try to do better than eve-marketdata.com, for example, is I will try to generate less internal server errors during downtime. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.24 22:30:00 -
[6]
Something which has absolutely nothing to do with any market data whatsoever, but I thought was needed anyway.
I've compiled a list of tools that everybody probably thinks are handy to have, but may not have yet heard about. I'll be working to keep that list up to date with everything new that might come out.
It can be found on: http://www.evemarketeer.com/home/tools
Hope you enjoy it. If you see your tool in the list, and wish to submit better information (logo, description, or if there's just something wrong), by all means, go ahead. Everything that's on the website now was simply taken from where I could find it, with the result that many applications do not have a logo, or an ugly one, or a rather ugly description. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.07.26 03:13:00 -
[7]
currently using your upload client, and it's using the upload ID that it seems to remember from the eve-marketdata client. Can you make it so you can edit the upload key on your page, so we can use the same key we use for eve market data, or alternatively, edit the upload client so it can accept one key for eve-marketdata and one key for evemarketeer? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.27 20:00:00 -
[8]
Originally by: Domi Etymology currently using your upload client, and it's using the upload ID that it seems to remember from the eve-marketdata client. Can you make it so you can edit the upload key on your page, so we can use the same key we use for eve market data, or alternatively, edit the upload client so it can accept one key for eve-marketdata and one key for evemarketeer?
I'd been trying to make my upload key the same as the one used for eve-marketdata, but after contacting its creator found out that it is impossible to duplicate it, as it's a unique key related to his database. Still, simply having two different fields is a pretty clean and simple idea. I'll get working on this. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.27 21:33:00 -
[9]
Edited by: Callean Drevus on 27/07/2011 21:35:05 Okey, the uploader has been updated, you are now able to enter both an EVE Marketeer upload key, and a EVE Marketdata upload ID seperately from another. I've also changed the header to be more in line with the application that the uploader is used for. Possible future upgrades are making it possible to specify an unlimited number of endpoints/upload keys.
I'm also happy to state that EVE Marketeer is now aggregating data from our own uploads, EVE Marketdata and EVE Central, about as realtime as I can make it. The only big thing missing now is an API that allows you to query all that data at once. But don't worry, that's coming soon ;)
Do you guys have any ideas of what should ABSOLUTELY be in the API? Or something you are missing somewhere else right now? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.07.28 00:10:00 -
[10]
Could I suggest you put a download link to your uploader somewhere on the front page. Either I'm utterly blind, or it's currently a real pain in the butt trying to find it! |
|
Domi Etymology
|
Posted - 2011.07.28 00:22:00 -
[11]
Just downloaded the uploader from http://www.evemarketeer.com/downloads/latest_client.zip
it's giving me NEMIA Uploader version 2.4EM
Is this the new uploader? If so.... where do you put in the 2nd new key for evemarketeer? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.28 11:19:00 -
[12]
I'm sorry, you are correct in your assumption that something went wrong. I'm working on fixing that now (I logically already had the new uploader on my PC, so it hadn't occurred to me yet to try to download it and see if it works). --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.28 14:31:00 -
[13]
Edited by: Callean Drevus on 28/07/2011 14:31:32 I'm happy to announce we now also provide an API for retrieving all the consolidated data from EVE Central, EVE Marketdata and our own uploads (only aggregated data/statistics for now). I calculate the following stats for all items in every region (and a specific set for the Jita station, which can be retrieved with a unique call):
type_id: the type id for which this data applies region_id: the region for which this data applies datetime: the time this data was generated sell_lowest: the lowest price this item is sold for sell_avg: the (weighted) average price this item is sold for sell_geo_mean: the geometric mean of the price this item is sold for sell_harm_mean: the harmonious mean of the price this item is sold for sell_highest: the highest price this item is sold for sell_lowest5: the average price of the lowest 5% of the sell market sell_orders: the amount of sell orders supply: the total amount of items available sell_worth: the cost of buying the entire sell market buy_highest: the highest price this item is bought for buy_avg: the average price this item is bought for buy_geo_mean: the geometric mean of the price this item is bought for buy_harm_mean: the harmonious mean of the price this item is bought for buy_lowest: the lowest price this item is bought for buy_highest5: the average price of the top 5% of the buy market buy_orders: the amount of buy orders demand: the total amount of items bought buy_worth: the profits of selling to the entire buy market
Go to http://www.evemarketeer.com/api to make use of this service. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Azazel Mordred
Minmatar Cloak of Shadows
|
Posted - 2011.07.28 17:06:00 -
[14]
Wow, your API looks really neat. Just the type of thing I've been looking for :). It's also fantastic that it's combining both Eve Central and Eve Marketdata in an easy to consume format.
I think this would make an excellent replacement for OOE's current market lookup (EVE-Central only).
Thanks!
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.29 22:53:00 -
[15]
Thank you. I'm happy to provide.
That said, I've added a trade route finder. If everything works correctly, it also filters out the idiotic/unrealistic orders. The only problem for now is it's exceptionally slow and locks up most of the server when it's calculating, but that will be fixed later :P just don't all start using it at the same time now. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.07.30 01:40:00 -
[16]
Something I've noticed, if you put in a partial item name into the search, ie: "trit" and then click on the "Tritanium" link from the dropdown, it fills the search box with Tritanium, but actually sends you to http://www.evemarketeer.com/item/itemprice/trit and therefore you get no results, since there is no "trit" item.
hope that made sense :P
That's using firefox, btw. Haven't tested in IE/chrome |
Almir Kadric
|
Posted - 2011.07.30 10:59:00 -
[17]
It's quite a funny thing actually, I've also been developing a similar tool but more focused around manufacturing to start. I've just about finished the manufacturing tool. So far I've spent a few days making my site.HTTP://EVECQ.COM
Manufacturing tool should be up either tomorrow or day after, just adding last pieces to it.
And what makes it funny is that since I've started (my tool started as a spreadsheet a few weeks ago till corpie asked for share, so I started web tool) is that I've noticed a lot of web tools popping up on the forums lately. Competition is going to be tough XD
|
Almir Kadric
|
Posted - 2011.07.31 09:21:00 -
[18]
Well I notice someone looked at my site after I posted it here, so to do myself some justice (though the website is far from complete) I've put up the new manufacturing tool. Take another look. Will start my own thread somewhere down the track once I'm happy with the volume of content and features.
SORRY TO THE OWNER OF THIS THREAD FOR SLIGHTLY HI-JACKING HIS THREAD XD
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.31 10:04:00 -
[19]
Edited by: Callean Drevus on 31/07/2011 10:09:53 @Domi: That should be fixed now. It's using the entire name for autocomplete too.
Ha! This is a free bump! I do not mind at all. It's quite funny to see how we all though of the same things (more or less) and started building it. I must say your manufacturing tool is exactly what I had in mind but were not able to build yet because I was focused on different things.
At least that website looks decent though! I'm sorry to say but you've given me some good ideas that I might steal :P alternatively, I'm starting to wonder whether it might not be better to work together on something epic instead of work parallel to eachother making everything that much slower to develop, but I also guess every one of us like his own baby. I for one wouldn't give up the domain name evemarketeer.com
Aside from that, your calculate button should be moved, and redesigned, I had trouble spotting it :P same for the ME and PE input fields. Am I correct in my assumption that it takes the most efficient numbers automatically? Oh, and you need a totals row for the materials cost (even though it's already in the top table. I'm also a bit confused as to what the difference between market and database is, since there seems to be none. Anyway, great website!
As for myself. -'ve been trying to see whether my trade route calculation fails to be fast because I am using MySQL (eve central does the same joins, but ends up incredibly faster than my website on PostgreSQL). If so, I'll probably be trying to port the entire website over to Postgres, since whatever I'm calculating I'm doing incredible amounts of joins (and inserts), and it seems postgre is good at that. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.07.31 14:45:00 -
[20]
Originally by: Callean Drevus Ha! This is a free bump! I do not mind at all. It's quite funny to see how we all though of the same things (more or less) and started building it. I must say your manufacturing tool is exactly what I had in mind but were not able to build yet because I was focused on different things.
Good to hear XD
Originally by: Callean Drevus At least that website looks decent though! I'm sorry to say but you've given me some good ideas that I might steal :P alternatively, I'm starting to wonder whether it might not be better to work together on something epic instead of work parallel to eachother making everything that much slower to develop, but I also guess every one of us like his own baby. I for one wouldn't give up the domain name evemarketeer.com
I agree on both points, the dilemma =S I'm the same with evecq.com such a nice simple name, yet capitalizes on eve's newest feature, surprised it was available XD
Originally by: Callean Drevus Aside from that, your calculate button should be moved, and redesigned, I had trouble spotting it :P same for the ME and PE input fields. Am I correct in my assumption that it takes the most efficient numbers automatically? Oh, and you need a totals row for the materials cost (even though it's already in the top table. I'm also a bit confused as to what the difference between market and database is, since there seems to be none. Anyway, great website!
Thanks for the feedback! To be honest i haven't put too much thought in the visual look. Just roughly did how it should look.
The ME and PE auto fill is a little better than just the most efficient numbers, it does 2 things. Calculates perfect ME/PE and then calculates ME/PE after 1 month of training and then picks the lower number. This way I don't get unrealistic ME/PE when i do my mass calculation caching.
Originally by: Callean Drevus As for myself. -'ve been trying to see whether my trade route calculation fails to be fast because I am using MySQL (eve central does the same joins, but ends up incredibly faster than my website on PostgreSQL). If so, I'll probably be trying to port the entire website over to Postgres, since whatever I'm calculating I'm doing incredible amounts of joins (and inserts), and it seems postgre is good at that.
Since you helped me, I'll give you something to help ;p
Don't bother migrating to posgres. I also did some crazy queries which too up to 10 minutes to complete! then I got them down to less than 100ms! (a page which did my special production calculations on every item in the game)
There's a few things to consider: 1. Indexes! (you need to index appropriate fields to help boost find times) 2. Caching IS GOLD (create new caching tables which map all possible options on the fly so you don't redo the same calculations over and over again) 3. Revise your SQL statements to be more efficient, use ANALYZE or just manually chunk out parts of the Joins and see what the difference in time is, once you locate your offending statement, optimize it.
If you need more help, shoot it my way. And I'm sure we can work together in some parts, since you're focusing on market aggregation (or so I'm assuming). Whilst I'm focusing on everything else where market aggregation is just but a small part. I'm sure if there's any sections of my site you like (such as my very complicated manufacturing tool or profit loss calculations on large scales) I'm sure I could provide you with an API embed you can use rather than writing all that grueling code yourself, not to mention wasted server CPU ;p
NOW OFF TO IMPLEMENT FEEDBACK PAGES along with your recommendations ;p Just got my news feed up. Thankful I finally got some more content on the home page Was looking quite sad =(
|
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.07.31 19:14:00 -
[21]
Originally by: Almir Kadric Since you helped me, I'll give you something to help ;p
Don't bother migrating to posgres. I also did some crazy queries which too up to 10 minutes to complete! then I got them down to less than 100ms! (a page which did my special production calculations on every item in the game)
There's a few things to consider: 1. Indexes! (you need to index appropriate fields to help boost find times) 2. Caching IS GOLD (create new caching tables which map all possible options on the fly so you don't redo the same calculations over and over again) 3. Revise your SQL statements to be more efficient, use ANALYZE or just manually chunk out parts of the Joins and see what the difference in time is, once you locate your offending statement, optimize it.
If you need more help, shoot it my way. And I'm sure we can work together in some parts, since you're focusing on market aggregation (or so I'm assuming). Whilst I'm focusing on everything else where market aggregation is just but a small part. I'm sure if there's any sections of my site you like (such as my very complicated manufacturing tool or profit loss calculations on large scales) I'm sure I could provide you with an API embed you can use rather than writing all that grueling code yourself, not to mention wasted server CPU ;p
The problem here is not so much in the way the queries are optimized (though I have to admit, with less joins, it works better ;)) and there's certainly optimization and pre-calculation possible for some of the things (if not a lot), but the most important part, matching sell orders against buy orders is just an insanely resource espensive part, even without all the other joins it takes ages to complete. All optimization I could throw at it didn't help much. I did manage to half the time needed by neglecting to sort though (doing that in PHP afterwards proved quite much faster than doing it in MySQL, which is pretty stupid).
There are probably some ways in which the query can be made better, but to be honest, I'm just interested in seeing how a different db turns out :) I won't be moving the site itself over until I see that the queries are actually running better. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Wind Jammer
Minmatar Molden Heath Software Company
|
Posted - 2011.07.31 21:30:00 -
[22]
Edited by: Wind Jammer on 31/07/2011 21:31:44
Originally by: Callean Drevus
The problem here is not so much in the way the queries are optimized (though I have to admit, with less joins, it works better ;)) and there's certainly optimization and pre-calculation possible for some of the things (if not a lot), but the most important part, matching sell orders against buy orders is just an insanely resource espensive part, even without all the other joins it takes ages to complete. All optimization I could throw at it didn't help much. I did manage to half the time needed by neglecting to sort though (doing that in PHP afterwards proved quite much faster than doing it in MySQL, which is pretty stupid).
There are probably some ways in which the query can be made better, but to be honest, I'm just interested in seeing how a different db turns out :) I won't be moving the site itself over until I see that the queries are actually running better.
UPDATE: There is a DISTINCT time advantage in using PostgreSQL. That is after spending a day trying to optimize MySQL as best as possible, then simply copying the database over into a default PostgreSQL install and running the query. That which didn't even complete within 2 minutes on MySQL takes about 15 seconds now. The only problem is migrating the entire codebase of the website :S
The best advice I can give you is to slice and dice the data you access online. The first incarnation of the EVE Production Mixer used the static dump as-is in a JavaDB database, and the SQL queries were pretty slow. I wrote an extract that created highly-prepared tables with had the data in a way that was convenient.
Now the Prod Mixer is hosted on Google App Engine which only supports a NoSQL datastore, and it's faster still :) If you do Java or Python check it out the GAE.
-edit Smart looking site btw :)
|
Almir Kadric
|
Posted - 2011.08.01 00:45:00 -
[23]
Originally by: Callean Drevus UPDATE: There is a DISTINCT time advantage in using PostgreSQL. That is after spending a day trying to optimize MySQL as best as possible, then simply copying the database over into a default PostgreSQL install and running the query. That which didn't even complete within 2 minutes on MySQL takes about 15 seconds now. The only problem is migrating the entire codebase of the website :S
To be honest the differences between posgres sql and mysql these days shouldn't be too big, if anything maybe a second or so. And this really sounds like a optimization issue to me. Mind showing me your SQL statement and letting me play with it? If I get it running smoothly on my DB it could save you a day or few hours (depending on what framework you're using).
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.01 06:57:00 -
[24]
Edited by: Callean Drevus on 01/08/2011 06:59:00
Originally by: Almir Kadric To be honest the differences between posgres sql and mysql these days shouldn't be too big, if anything maybe a second or so. And this really sounds like a optimization issue to me. Mind showing me your SQL statement and letting me play with it? If I get it running smoothly on my DB it could save you a day or few hours (depending on what framework you're using).
You are welcome to look at them. The easiest way to get the tables to try out this query is getting them from eve-marketdata.com, they provide a dump of all the buy/sell orders in all regions, and they're the most important tables to use for this query.
This is what I'm doing (in basis, the true query has more joins, but those are not tables easily reproduced :) SolarSystemDistances contains a table with distances between all solar systems. You can remove it without having too big an impact on the overal time taken. All fields which are mentioned in joins &| where have been indexed.
SELECT ise.type_id, ise.qty_avail AS from_qty, ibu.qty_avail AS to_qty, IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) AS transferrable, ise.price as sell_price, ibu.price as buy_price, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail),0) as total_profits, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) / ssd.Distance,0) as profit_per_jump, ssd.Distance FROM items_selling ise INNER JOIN items_buying ibu ON ibu.station_id = 60003760 AND ise.type_id = ibu.type_id AND ise.qty_min <> ise.qty_avail INNER JOIN SolarSystemDistances ssd ON ssd.FromSolarSystemID = ise.solarsystem_id AND ssd.ToSolarSystemID = ibu.solarsystem_id WHERE ise.date_created > NOW() - INTERVAL 8 HOUR ise.station_id = 60008494 AND ise.price < ibu.price LIMIT 20;
If you can change this query in any way to make it useful, I'd be very much obliged. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Ila Minn
|
Posted - 2011.08.01 08:44:00 -
[25]
First of all: Thanks for providing service to the community.
I very much like the concept of how you both aggregate and share data, and would very much like to make more use of the data you provide. Sadly, you don't seem to provide any method to download orders for items in regions, which makes it hard to evaluate two things that I would be very keen on knowing:
- The highest buy/lowest sell price of an item in a station other than Jita. - The number of units of an item available _close_ to the highest buy/lowest sell price for a given station (like, within one or two percent of the price).
Maybe you could extend the market aggregation data to at least contain the second information for the Jita prices, that in itself would already make me a happy minmatar.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.01 10:39:00 -
[26]
Edited by: Callean Drevus on 01/08/2011 10:41:24
Originally by: Ila Minn First of all: Thanks for providing service to the community.
I very much like the concept of how you both aggregate and share data, and would very much like to make more use of the data you provide. Sadly, you don't seem to provide any method to download orders for items in regions, which makes it hard to evaluate two things that I would be very keen on knowing:
- The highest buy/lowest sell price of an item in a station other than Jita. - The number of units of an item available _close_ to the highest buy/lowest sell price for a given station (like, within one or two percent of the price).
Maybe you could extend the market aggregation data to at least contain the second information for the Jita prices, that in itself would already make me a happy minmatar.
Since making happy minmatars is one of the goals of this operation, I've added the ability to do both. You can query aggregated data for any station now (calculated on the fly, so you're always sure I'm using the most recent orders) and I've added 2 new fields to the data which show the amount of items available within 5% of the lowest sell/highest buy price. Since these are calculated when orders are uploaded (or requested via the new API call), it might take a bit before it's propagated across all items/regions.
I'm sorry for the location of the fields (in the middle of the data, meaning if you already did something with CSV parsing it would now be messed up), but fields are exported in the same order as they are in my database, and I have to prevent myself from going insane. After another week or so I'll make a distinct API version that will not change afterwards anymore (can continue to be used without any significant changes except bugfixes). --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 04:43:00 -
[27]
Originally by: Callean Drevus Edited by: Callean Drevus on 01/08/2011 10:43:51
Originally by: Almir Kadric To be honest the differences between posgres sql and mysql these days shouldn't be too big, if anything maybe a second or so. And this really sounds like a optimization issue to me. Mind showing me your SQL statement and letting me play with it? If I get it running smoothly on my DB it could save you a day or few hours (depending on what framework you're using).
You are welcome to look at them. The easiest way to get the tables to try out this query is getting them from eve-marketdata.com, they provide a dump of all the buy/sell orders in all regions, and they're the most important tables to use for this query.
This is what I'm doing (in basis, the true query has more joins, but those are not tables easily reproduced :) SolarSystemDistances contains a table with distances between all solar systems. You can remove it without having too big an impact on the overal time taken. All fields which are mentioned in joins &| where have been indexed (though they have all been indexed separately, would it be better if they formed a single index?).
SELECT ise.type_id, ise.qty_avail AS from_qty, ibu.qty_avail AS to_qty, IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) AS transferrable, ise.price as sell_price, ibu.price as buy_price, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail),0) as total_profits, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) / ssd.Distance,0) as profit_per_jump, ssd.Distance FROM items_selling ise INNER JOIN items_buying ibu ON ibu.station_id = 60003760 AND ise.type_id = ibu.type_id AND ise.qty_min <> ise.qty_avail INNER JOIN SolarSystemDistances ssd ON ssd.FromSolarSystemID = ise.solarsystem_id AND ssd.ToSolarSystemID = ibu.solarsystem_id WHERE ise.date_created > NOW() - INTERVAL 8 HOUR ise.station_id = 60008494 AND ise.price < ibu.price LIMIT 20;
If you can change this query in any way to make it useful, I'd be very much obliged.
You're such an ASS XD LOL your problem drove me a little nuts for the wrong reasons. Let me explain ;p I first wrote a piece of SQL to get the shortest route between 2 systems and the number of jumps HOWEVER as we both know the math in that is insane and it was slow. So then i began to engineer a caching system shortest jump distances, but the permutations are insane. So i went a step further and created a script which does it in the background, without overlaps increasing the speed quite nicely and fixing all bugs and glitches.
SO with the cache table being available to me I ran you SQL and found it's problem, only took me a few seconds lol.
the problem is the 2 following statements: 1. INNER JOIN items_buying ibu ON 2. ise.price < ibu.price
The first obviously since it's trying to match all the data across an insane amount of rows.
The second is actually a funny point because it's a math check, which you'd think is fast, however in a database select algorithm it's extremely CPU intensive and time consuming, since it checks every row of data this also applies to the above comment.
NOW THE IRONY, it's actually faster to sort the data (KEY POINT). Since it just has to bubble sort it as it spits it out.
THUS THE NEW QUERY....WILL CONTINUE ON NEXT POST (DAMN CHARACTER LIMIT)
|
Almir Kadric
|
Posted - 2011.08.02 04:48:00 -
[28]
Originally by: Almir Kadric THUS THE NEW QUERY....WILL CONTINUE ON NEXT POST (DAMN CHARACTER LIMIT)
Stupid 2 minutes between posts time limit as well =_=
HERE IT IS:
SELECT ise.type_id, ise.qty_avail AS from_qty, ibu.qty_avail AS to_qty, IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) AS transferrable, ise.price as sell_price, ibu.price as buy_price, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail),0) as total_profits, FORMAT((ibu.price - ise.price) * IF(ise.qty_avail > ibu.qty_avail, ibu.qty_avail, ise.qty_avail) / ssd.jumps,0) as profit_per_jump, ssd.jumps, ise.date_created FROM `eve-marketdata_items_selling` AS ise INNER JOIN `eve-marketdata_items_buying` AS ibu ON ibu.type_id = ise.type_id LEFT JOIN `eve_jumpsbetweensystems` ssd ON ssd.fromSystemID = ise.solarsystem_id AND ssd.toSystemID = ibu.solarsystem_id WHERE ise.station_id = 60008494 AND ibu.station_id = 60003760 AND ise.qty_min <> ise.qty_avail AND ise.date_created > (NOW() - INTERVAL 55 HOUR) ORDER BY profit_per_jump DESC LIMIT 20
EXECUTES IN .218 seconds ^_^
Now I could give you the exact math as to why this happens (algorithm execution time math by instruction order) but I honestly CBF opening my OLD uni text book to look it up, and it's been a while since I did the math so I can't remember it (something like ln(n) per loop instruction blah blah)
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 10:35:00 -
[29]
Originally by: Almir Kadric EXECUTES IN .218 seconds ^_^
Now I could give you the exact math as to why this happens (algorithm execution time math by instruction order) but I honestly CBF opening my OLD uni text book to look it up, and it's been a while since I did the math so I can't remember it (something like ln(n) per loop instruction blah blah)
UPDATE: sorry for mass updates, didn't know the right code tag XD
You are my hero!
Though it still takes about 8 seconds+ to finish the query, at least it actually finishes within a reasonable amount of time now. After caching it's fast indeed. If it's fast on your first query I'm probably going to have to do some optimization to my MySQL server ;)
I don't think I'd have thought about the price < price statement being the issue... As you said, math should be fast, even for lots of rows (you're doing that math too if you're sorting) still, I'll keep this one in mind in case something like this ever happens again. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 10:54:00 -
[30]
Edited by: Almir Kadric on 02/08/2011 10:55:07
Originally by: Callean Drevus
You are my hero!
Though it still takes about 8 seconds+ to finish the query, at least it actually finishes within a reasonable amount of time now. After caching it's fast indeed. If it's fast on your first query I'm probably going to have to do some optimization to my MySQL server ;)
Not sure if it's fast on my first query, but since you should only realistically have 1 first query every hour at most (once a day for me since im using static dumps at the moment) I don't see it as that much of an issue. One user takes the **** for everyone else's benefit
Originally by: Callean Drevus
I don't think I'd have thought about the price < price statement being the issue... As you said, math should be fast, even for lots of rows (you're doing that math too if you're sorting) still, I'll keep this one in mind in case something like this ever happens again.
Well I could tell from one look and let me explain why, it should help you understand DB's better. When data is stored in a DB it's stored within a type of tree which is sorted by the primary keys of that table. And in certain cases when you reach a leaf that leaf could be a bucket of data which is once again sorted. So when you traverse a database tree, to get ordered items is easy as it just traverses the tree differently and worst case scenario it would have to sort the smaller bucket. BUT if you take math logic into the picture which a DB isn't sorted by it has to execute the math then resort the entire resultant set. Now this is purely hypothetical and situation based (DB type, engine etc). BUT the moral here is avoid anything that isn't already in the db as a key or resultant set, unless the set you get back is quite small.
Originally by: Callean Drevus
Also, I'm sorry for not providing the SolarSystemDistances table, if I could have remembered exactly WHERE I had found it I would happily have provided...
So you actually got someone elses data for this XD, I looked for it and couldn't find it. I'm slowly rebuilding my own set atm, if you do find where you got it from would appreciate it heaps. Otherwise I'll quietly work on a new piece of code which also can detect defects in shortest paths when i have to update the EVE Static Datadump as the mapsolarsystemjumps table could change. You gave me quite a side project to accomplish XD
Let me know if you have any other issues ^_^
|
|
Almir Kadric
|
Posted - 2011.08.02 10:58:00 -
[31]
BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?
And does it provide the actual route between those systems?
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 11:08:00 -
[32]
Originally by: Almir Kadric BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?
And does it provide the actual route between those systems?
AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.
Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 11:23:00 -
[33]
Edited by: Almir Kadric on 02/08/2011 11:28:43
Originally by: Callean Drevus AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.
So it shows all routes and not just the shortest? That's how I did my original caching, but I've changed it since to only do shortest. Also I changed mine to only cache high sec systems since realistically not many would actually trade through low or null sec. And it shows the actual route, checked it against actually in game Auto Pilot so far identical. Only down side is i recently did the caching for 10-15 jump routes and it took 4 hours =_= currently figuring out a way to optimize it further, otherwise ill have to whip out my super computer in a box to do the cache once and for all then just save it somewhere.
Originally by: Callean Drevus Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas?
EDIT: Just read your previous post edit. I see. That's unusual. I tested it on mine again and it seems fine. Did you change anything? show me your full query and ill take a look.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 12:15:00 -
[34]
Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields
There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 13:38:00 -
[35]
Originally by: Callean Drevus Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields
LOL didn't know that
Originally by: Callean Drevus There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so.
I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.
And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 14:13:00 -
[36]
Originally by: Almir Kadric I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.
And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.
Well, I try to have my marketdata as realtime as possible, so when someone uploads something new, I want those orders in my database as fast as possible.
Of course, I'm not calculating the traderoutes again everytime I get new data, that would be database suicide :) I just don't think the MySQL query cache likes it when I constantly add new data to mess up it's calculations. But I have not yet put serious research into that.
On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.
I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 19:13:00 -
[37]
Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.
I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?
It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.
I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent... --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.03 02:34:00 -
[38]
Originally by: Callean Drevus On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.
Glad to hear it ^_^
Originally by: Callean Drevus I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z
LOL thanks for this, ill check what it contains and decides what's better, the one i built or this one.
Originally by: Callean Drevus
Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.
I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?
It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.
I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent...
Hmm sounds like eve-marketdata is caching the information first then only allowing exports after a certain time period. For example even though they get data in real time. They probably have that in a table which is only for inserts. Then they have a separate table which is built every hour or what ever time period which is specifically for select statements, which makes sense since they can speed up those statements, not to mention if they run a first run SQL cache statement to have sql ready for fast reads and just hot-swapping the databases over when they're ready.
Best to send them an email to find out what they're up to ;p
Let me know if you find anything
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.03 20:14:00 -
[39]
For everyone who is interested. The market scanner of evemarketeer is now able to scan only items you enter manually, therefore scanning only the items that are of interest to you. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.04 00:17:00 -
[40]
With the default upload suggestions for the market scanner, could I suggest you take out the Tech 2 BPO's, since I'm 99.999999% sure they aren't sold on the market anymore ;) |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.04 09:56:00 -
[41]
It seems there IS a marketgroup for them though :P as that is what I am filtering by. I'll look into removing them ;) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Temuil
|
Posted - 2011.08.05 15:09:00 -
[42]
I've been using this for a few days and liking it already.
I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.05 20:48:00 -
[43]
Originally by: Temuil I've been using this for a few days and liking it already.
I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added?
Jup, that is a legacy part of the website from when I was only using it myself. The other regions have been added/will soon appear. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.05 21:01:00 -
[44]
I almost forget that I have added a few new features.
You can now look up traderoutes within and across a few empire regions (these are updated every hour, you can see when they will be updated, so if you check a few minutes after that, they should be fresh). Provided data is actually uploaded in the regions where you try to find trades :) Trades older than 24 hours cannot be found, as it makes no sense, they'll be fulfilled anyway (I think, please contradict me if I'm wrong).
For now, eve central orders are not taken into account when calculating trades, since any and all orders I got during testing were either fake or outdated (even if updated 5 minutes previously). Didn't seem like a good idea.
You can also create a list of items to check their total value (and export and save the list to be used again later). Eventually saved on server, for now, you'll have to create your own text file :) I used this one to check the total value of items in a contract. Until CCP releases the Contracts API. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.07 19:58:00 -
[45]
I just updated evemarketeer with a feature that I think traders will love. It already exists on eve-marketdata of course, but I've streamlined the process a bit.
It involves checking whether your orders are really the lowest in the region. All updating of data is done via ajax, so the only thing you have to do is periodically check whether all the rows are still green.
Check it out at http://www.evemarketeer.com/trade/orders --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.07 23:04:00 -
[46]
In addition, I have an interesting image here, of what happens when you get your data from 3 different sources that for some idiotic reason do not align.
I'm talking about my own data, eve-marketdata and eve-central. Does anyone see a pattern here: http://www.evemarketeer.com/graph/tiny_history/10000043/20413/2? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Cleuseau Remos
|
Posted - 2011.08.08 15:55:00 -
[47]
Callean,
Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.
Example would be: http://www.evemarketeer.com/api/jita_info/34/xml
Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.
TIA!
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 00:38:00 -
[48]
Originally by: Cleuseau Remos Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.
Example would be: http://www.evemarketeer.com/api/jita_info/34/xml
Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.
I couldn't leave this alone :P
Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success). --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.09 02:28:00 -
[49]
Originally by: Callean Drevus
I couldn't leave this alone :P
Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success).
Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 09:29:00 -
[50]
Edited by: Callean Drevus on 09/08/2011 09:29:36
Originally by: Almir Kadric Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.
Actualy, I'm building the XML myself, since there'a always only a set of rows to display it I really didn't think about using the php DOM or XML functions. There are very little ways in which XML can not be ready for xpath use if it's well formatted. And given the depth of my XML, it's pretty much impossible to make any mistake.
What I was talking about when I said 'changes to the way data is presented' I meant made sure that a 'content-type: text/xml' header is sent, which google docs seems to like :)
PS: It is however, not impossible that I'll fall back to PHP lib functions if the output gets any more complex :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 20:32:00 -
[51]
People, I'd like to present my awesome feedback and prioritization form, allowing you to make your wishes known more easily. Please use it! --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.10 01:09:00 -
[52]
Eve's randomly down at the moment, so can't test the feedback form in the game browser, but trying to use it in firefox gives me "javascript:$('#descr_5').toggle();" in the address bar and "[object Object]" on the web page. Did I derp? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.10 13:10:00 -
[53]
Nope, you didn't derp, I did. Seems firefox doesn't like me putting javascript function in the href anymore. In any case, it should be fixed now. It's really unhandy if there's a bug in the feedback form :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.11 12:10:00 -
[54]
Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.
For little bugs like this, do you want us to post here, or on the feedback thing you made? |
Turhan Bey
|
Posted - 2011.08.11 15:20:00 -
[55]
- Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.
Error: EVE Marketdata upload key must be set before upload! Uploaded 38 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 35 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 34 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 39 in Sinq Laison
- What exactly determines the amount of "credits" given in the Top Uploaders list?
Top Uploads, Turhan Bey: 174,723 Top First Uploads, Turhan Bey: 70,787 Top Credits, Turhan Bey: 1,601,930.60
Top Uploads, Temuil: 72,783 Top First Uploads, Temuil: 10,251 Top Credits, Temuil: 1,716,783.58
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.11 17:36:00 -
[56]
Edited by: Callean Drevus on 11/08/2011 17:48:33
Originally by: Domi Etymology Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.
For little bugs like this, do you want us to post here, or on the feedback thing you made?
Both are fine, if you can put it on feedback I will not forget about if the amount of posting here will suddenly increase for one reason or another.
Originally by: Turhan Bey - Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.
- What exactly determines the amount of "credits" given in the Top Uploaders list?
I see, this probably happened when you turned it off while the program was working. It was not updating a required variable correctly, leaving it in the previous state until restart.
In regards to the way the credits are calculated. I is based on how much orders have changed since the last upload, and on how much time has elapsed between then and now. It also takes into account whether there have been previous uploads for that item (hence the first upload thing), because if I did not, you would get MASSIVE amounts of credits when uploading information for an item for the first time (actually, I think that would grant you infinite credits ). What I think is that Temuil must have uploaded data that was very much changed since before, in regions that were very much outdated. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.11 20:27:00 -
[57]
I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.11 20:38:00 -
[58]
Originally by: Turhan Bey I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.
Is the checkbox itself in the correct state after restarting? I'll have this fixed in the next uploader version. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.11 21:59:00 -
[59]
Yep, the checkbox itself appears correct. But the uploader behaves as if it were still selected.
|
Turhan Bey
|
Posted - 2011.08.12 16:20:00 -
[60]
re: credits
I assume that you have an upper cap on the time for repeated submissions, probably along the lines of a few days or a week. Otherwise, remote locations that may be ignored for many months at a time would generate credits also at huge levels. I would recommend treating a first-time submission at this cap level, i.e. reporting an item for the first time is equivalent to being the first to report it in N weeks/months at whatever is your cap.
Otherwise, what you are essentially telling your users is that it is more valuable for them to not go out to new areas and scan them, but instead repeatedly scan existing areas. It is also not valuable to keep remote area prices up-to-date, and they instead should wait as long as possible before going back and submitting updated info. |
|
Callean Drevus
Caldari New Eden Law Enforcement
|
Posted - 2011.08.12 18:12:00 -
[61]
Turhan Bey, you are correct. I had the credits for new uploads at a limited value at first, since I expected to get uploads from the easy regions a lot at first (leading to massive amounts of credits for fairly easy work, which is a bit opposed to the purpose), I will be changing this to be as you say soon (maximum credits for never before uploaded items), since I figure most easy uploads have been done by now. The time factor on uploads is not so high as to make it very profitable to wait a long time before uploading new data, but repeatedly spamming the upload of a certain item won't work either. Then again, if you upload an item very often and the changes are huge (tritanium has a tendency to have many changed orders, even in just half an hour) it might actually pay off more than going out to remote regions and uploading new and/or less often updated prices. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari New Eden Law Enforcement
|
Posted - 2011.08.12 22:10:00 -
[62]
Edited by: Callean Drevus on 12/08/2011 22:10:37 On a different note, feedback has been upgraded! You can now comment and I can mark items as complete without breaking the whole system.
I've also made progress on implementing inferred orders due to a complete rewrite of the system used for importing orders (all different sources were using different ways to import data up until now, so I essentially had the same code thrice). If that part is finished, I'll be able to start on building features in earnest :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.14 17:05:00 -
[63]
Version 2.5.1 seems to have fixed the E-MD upload checkbox issue, thank you. :)
Could you also make a download available that is not wrapped in an MSI package? Just a ZIP like the old 2.5 download, basically. Or has the format changed such that an install is needed?
|
Callean Drevus
Caldari New Eden Law Enforcement
|
Posted - 2011.08.14 17:43:00 -
[64]
Edited by: Callean Drevus on 14/08/2011 17:47:12
Originally by: Turhan Bey Version 2.5.1 seems to have fixed the E-MD upload checkbox issue, thank you. :)
Could you also make a download available that is not wrapped in an MSI package? Just a ZIP like the old 2.5 download, basically. Or has the format changed such that an install is needed?
Good :) and no, nothing has changed significantly that makes the MSI necessary, it's just that the installer is able to make the program autostart, which you might regard as both good or bad (I personally like it). I'll happily provide a zip download as well.
Will soon put up a download page where you can get the msi or zip version of the uploader as well as historical downloads, just in case I mess up sometime.
Everyone might also notice that I've modified the credit calculation algorithm, leading to a better distribution of overal credit for uploads. I'm sorry for our previous record holder, but it seems this changed the distribution of credits significantly
More features coming in a new release tonight! --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.15 02:25:00 -
[65]
I'm finding an interesting situation with the uploading client. If I'm running to eve clients dumping data, it goes fine. However if I add a third client, the uploader seems to freeze up after a few seconds. It sort of looks like it can't handle the speed that items are entering into eve's cache, or something.
Anyone else noticed anything similar, or am I the only nut who tries market dumping data with 3 or more clients at once :P |
Callean Drevus
Caldari New Eden Law Enforcement
|
Posted - 2011.08.15 09:58:00 -
[66]
Originally by: Domi Etymology I'm finding an interesting situation with the uploading client. If I'm running to eve clients dumping data, it goes fine. However if I add a third client, the uploader seems to freeze up after a few seconds. It sort of looks like it can't handle the speed that items are entering into eve's cache, or something.
Anyone else noticed anything similar, or am I the only nut who tries market dumping data with 3 or more clients at once :P
I must confess my testing was limited at two clients, though I fail to see how three clients shouldn't work. I'll work on finding out though ;) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.15 15:15:00 -
[67]
Same, I've never run more than two clients simultaneously, and usually only run one. Unfortunately, I don't think that I'll be able to help test this one because I don't believe that my system could handle more.
|
Domi Etymology
|
Posted - 2011.08.16 03:14:00 -
[68]
Hasn't stopped you from uploading a ton of info though :) Do you leave two clients going almost constantly anytime you're not actively using your account? |
Dragonaire
Caldari Corax. PURgE Alliance
|
Posted - 2011.08.16 06:01:00 -
[69]
I was reading back through the thread after something a friend said about you having some problems with slow queries and thought I'd see if you'd looked into MariaDB and TokuDB instead of MySQL? Maria is a drop in replacement for MySQL and is faster and has less bugs. Toku is another DB engine like InnoDB or MyISAM which you can add on as a plugin and use. It uses a different index structure that makes it a lot faster with more indexes which is where the slow down usually is because you either have a lot and inserts become real slow or don't have them and pay with slow selects. I've tested Yapeal using them and even with very small datasets the queries were measurably faster.
Anyway if you're interested in talking about it some more contact me via my E-mail from Google code and I'll go into a little more details with you. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.16 08:41:00 -
[70]
Originally by: Dragonaire I was reading back through the thread after something a friend said about you having some problems with slow queries and thought I'd see if you'd looked into MariaDB and TokuDB instead of MySQL? Maria is a drop in replacement for MySQL and is faster and has less bugs. Toku is another DB engine like InnoDB or MyISAM which you can add on as a plugin and use. It uses a different index structure that makes it a lot faster with more indexes which is where the slow down usually is because you either have a lot and inserts become real slow or don't have them and pay with slow selects. I've tested Yapeal using them and even with very small datasets the queries were measurably faster.
Anyway if you're interested in talking about it some more contact me via my E-mail from Google code and I'll go into a little more details with you.
I happened to come across MariaDB before, in connection with the thread on this forum in regards to route processing, but I haven't given it much attention yet. I hadn't heard about Toku though, which might be interesting, even if it's only for the inserts, which is something that's happening all the time.
EVE Marketeer is running on my company production server though, and I'm not so hot on experimenting with new technologies when I can break several of my 'real' websites. I'll find out some more about them, and I'll contact you later. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
|
Dragonaire
Caldari Corax. PURgE Alliance
|
Posted - 2011.08.16 14:22:00 -
[71]
Good to hear you are giving them a try. Try the Yapeal project owner at Gmail and you'll find me -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.16 22:23:00 -
[72]
Ah, yes, I must be an idiot :P
For those industrialists here. I've added a feature that allows you to calculate how much resources building a certain amount of items will cost you in both time and resources. It also shows you how much profit you will make selling to current market prices. It's awesome! Give it a try. Just go to the page of a random item (that can be built) and look at the bottom of the manufacturing & research tab :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Johnathan Roark
Caldari The Graduates Morsus Mihi
|
Posted - 2011.08.16 22:53:00 -
[73]
Originally by: Callean Drevus Ah, yes, I must be an idiot :P
For those industrialists here. I've added a feature that allows you to calculate how much resources building a certain amount of items will cost you in both time and resources. It also shows you how much profit you will make selling to current market prices. It's awesome! Give it a try. Just go to the page of a random item (that can be built) and look at the bottom of the manufacturing & research tab :)
very neat tool, but im getting an error:
Warning: Missing argument 5 for mod_item::minerals(), called in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/item.php on line 228 and defined in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/item.php on line 201
Warning: Missing argument 5 for mod_item::minerals(), called in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/item.php on line 228 and defined in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/item.php on line 201
http://www.evemarketeer.com/item/itemprice/Absolution
POS-Tracker 3.0 Hosting |
Domi Etymology
|
Posted - 2011.08.17 06:46:00 -
[74]
This bug seems to be with anything not t1. Tested it out looking for a few t2 ships, and t3's. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.17 09:39:00 -
[75]
Whoops, forgot to add a new argument for my recursive functions, thats why you were seeing so MANY errors.
It should be fixed now. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.17 22:41:00 -
[76]
Edited by: Callean Drevus on 17/08/2011 22:42:21 Oh, I've been distracted from my previous goal by creating a nice page for tracking your wallet transactions. It creates a summary for transactions over the last x days, and allows you to filter your transaction list by type, station, and client (the last two of which are probably useless, but the first one might prove pretty useful).
It obviously requires you to enter you full API key and select some characters, but otherwise it should be pretty epic :)
http://www.evemarketeer.com/trade/transactions --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.18 01:52:00 -
[77]
Originally by: Callean Drevus EDIT: Ahrg, morons uploading false data to EVE Central are messing up my alghorithms now I've got a neat little project for the next few days. Finding a way to identify false uploads...
Would it be feasible to have a checkbox to allow whether the user wanted to include eve-central and/or eve-marketdata info in their results?
Or is this just a terribad-will-break-everything-and-cause-the-apocalypse sort of idea? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.18 08:19:00 -
[78]
Originally by: Domi Etymology
Originally by: Callean Drevus EDIT: Ahrg, morons uploading false data to EVE Central are messing up my alghorithms now I've got a neat little project for the next few days. Finding a way to identify false uploads...
Would it be feasible to have a checkbox to allow whether the user wanted to include eve-central and/or eve-marketdata info in their results?
Or is this just a terribad-will-break-everything-and-cause-the-apocalypse sort of idea?
It is not a terrible option, but due to the way the website is build, completely impossible all data is mixed and matched on upload, even though the information on who uploaded it is available, there is no way to retrieve the state of the orders from a different service (well, there is, but it's not something you should want to use for that)
Besides, I just cannot let myself be defeated by phonies. That is also the only reason my uploader is not open source, it prevents people from looking at the source (hopefully) and finding out the data format and endpoint for my uploads. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.18 23:38:00 -
[79]
Originally by: Domi Etymology Hasn't stopped you from uploading a ton of info though :) Do you leave two clients going almost constantly anytime you're not actively using your account?
Yeppers, I have a second system that is even lower-end (it will definitely choke and die if made to run dual clients), but it is a basic server system that is running 24/7. So whenever I am not online with one account or the other, then I try to use that system to log it in and just run a constant market scan, and when convenient from within a random (uncommon) region.
|
Salam Horani
|
Posted - 2011.08.20 08:48:00 -
[80]
hi callean,
great site you have there, thank you.
i have an issue, though. i don't seem to be able to add my API keys or add characters to my account. i get all sorts of error. posting bellow the errors i get when i try to add api key or characters:
Fatal error: Uncaught exception 'LogicException' with message 'Api call requires user credentials' in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php:187 Stack trace: #0 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php(211): AleEVEOnline->addCredentials(Array, 2) #1 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/util/context.php(60): AleEVEOnline->_retrieveXml(Array, Array) #2 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/player.php(102): AleUtilContext->__call('Characters', Array) #3 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/player.php(102): AleUtilContext->Characters() #4 [internal function]: mod_player->api(Object(Request)) #5 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/main.php(46): call_user_func_array(Array, Array) #6 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/index.php(11): include('/home/admin/dom...') #7 {ma in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php on line 187
|
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 10:03:00 -
[81]
Originally by: Salam Horani hi callean,
great site you have there, thank you.
i have an issue, though. i don't seem to be able to add my API keys or add characters to my account. i get all sorts of error. posting bellow the errors i get when i try to add api key or characters:
Fatal error: Uncaught exception 'LogicException' with message 'Api call requires user credentials' in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php:187 Stack trace: #0 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php(211): AleEVEOnline->addCredentials(Array, 2) #1 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/util/context.php(60): AleEVEOnline->_retrieveXml(Array, Array) #2 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/player.php(102): AleUtilContext->__call('Characters', Array) #3 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/mod/player.php(102): AleUtilContext->Characters() #4 [internal function]: mod_player->api(Object(Request)) #5 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/main.php(46): call_user_func_array(Array, Array) #6 /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/index.php(11): include('/home/admin/dom...') #7 {ma in /home/admin/domains/coronum.nl/public_html/dev/evemarketeer.com/core/lib/ale/eveonline.php on line 187
Hey Salam, are you by any chance using your account name as the account id? I'll take a look at this behaviour, I know my API additions are a little volatile, but I've refrained from changing it as of yet, since CCP will release the new API keys at the end of august anyway and I'll have to rewrite it completely then anyway. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Salam Horani
|
Posted - 2011.08.20 10:08:00 -
[82]
Originally by: Callean Drevus
Originally by: Salam Horani hi callean, Hey Salam, are you by any chance using your account name as the account id? I'll take a look at this behaviour, I know my API additions are a little volatile, but I've refrained from changing it as of yet, since CCP will release the new API keys at the end of august anyway and I'll have to rewrite it completely then anyway.
no i'm not. first time i clicked on the api info link, i got a page where i could insert my api key. when i submitted the data, i got that error, and i've been getting it ever since. hope that helps.
cheers,
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 10:10:00 -
[83]
Ok, I'll take a look at it :)
What is your account name? It seems you are not registered under Salem? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Salam Horani
|
Posted - 2011.08.20 10:31:00 -
[84]
my account name is "mirage3d"
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 11:07:00 -
[85]
Salam, I have made a few adjustments to the way in which API keys were entered. It should work fine now. Seems for one reason or another your api key was inserted with a user id of 0 --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Salam Horani
|
Posted - 2011.08.20 13:01:00 -
[86]
hi Callean,
thanks for your fast reply and action. the page works fine now, but i think i know what went wrong. i probably input the "user id" that i used to register with your site instead of my "account number" on EVE. the same mistake happened twice and now it won't let me input the right data even though. i input my correct account number. now i get a confirmation about my preference in green followed by the error in red:
"Your character preferences have been saved. Data will be updated periodically for the selected characters."
"New API credentials invalid, this combination of key and user id does not exist."
hope that helps, and thanks again.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 14:04:00 -
[87]
Ok, I think it should be completely fixed now. I accidentally broke the whole registration process because of a typo. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Salam Horani
|
Posted - 2011.08.20 14:20:00 -
[88]
Edited by: Salam Horani on 20/08/2011 14:27:20
Originally by: Callean Drevus Ok, I think it should be completely fixed now. I accidentally broke the whole registration process because of a typo.
yup, it works now. thanks a lot. i still get "You have not added any characters yet. Try adding some?" at the "My Market" page even though i checked both characters that i play. when i click on the link "adding some" it takes me back to the "Accounts/API Details" page.
edit: it now magically lists both my characters, unless the magic was performed by you hehehehe. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 14:45:00 -
[89]
Nope, the magic is performed by yapeal. It takes a little while for it to update the character sheets, which are necessary. I believe the creator of yapeal is now working on a possibility to instantly query certain types of data, removing the delay you now noticed. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Karbowiak
Sniggerdly
|
Posted - 2011.08.20 18:20:00 -
[90]
It saddens me greatly you list Griefwatch, but not EVSCO or EVE-KILL.. Saddens me very greatly!
Co-Owner and Creator of EVSCO |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.20 22:43:00 -
[91]
Edited by: Callean Drevus on 20/08/2011 22:43:54
Originally by: Karbowiak It saddens me greatly you list Griefwatch, but not EVSCO or EVE-KILL.. Saddens me very greatly!
Then we'll just have to remedy that won't we?
Added EVSCO, since EVE-KILL is just a subdepartment of EVSCO.
UPDATE: Woooot, page 4, never thought I'd get this far ! --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Ila Minn
|
Posted - 2011.08.21 08:33:00 -
[92]
http://www.evemarketeer.com/api/recent_updates
returns:
error No result found
*sniffs* Don't want to browse through all your data to find what's new :)
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.21 08:43:00 -
[93]
Originally by: Ila Minn http://www.evemarketeer.com/api/recent_updates
returns:
error No result found
*sniffs* Don't want to browse through all your data to find what's new :)
Phew, it isn't as bad as I thought it could be :) data is still happily updating. I'll see why it isn't working as intended anymore. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Salam Horani
|
Posted - 2011.08.21 16:11:00 -
[94]
hi callean,
i can't seem to get the market scanner to work. i'm on a mac so i can't use your uploader. this is what i get when i browse the "market scanner" page:
"Uploader has no idea where you are. You will be able to build manual groups, but you will not be able to start scanning the market."
so i don't see the start button anymore.
p.s. any plans to release a mac version of your scanner?
|
Turhan Bey
|
Posted - 2011.08.21 18:00:00 -
[95]
I've downloaded the new "2.5.1EM" client, standalone zip version, and it seems to be working just fine. However, upon startup I receive:
Error: Version information incorrect: 2.51EM 2.5.1EM: Minor fixes & automated installer + startup http://www.evemarketeer.com/var/files/uploader-2.5.1EM.msi
This was what clued me in to the new version in the first place. :) Could there be an old registry setting that still holds the old version string or something?
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.21 18:47:00 -
[96]
Originally by: Turhan Bey I've downloaded the new "2.5.1EM" client, standalone zip version, and it seems to be working just fine. However, upon startup I receive:
Error: Version information incorrect: 2.51EM 2.5.1EM: Minor fixes & automated installer + startup http://www.evemarketeer.com/var/files/uploader-2.5.1EM.msi
This was what clued me in to the new version in the first place. :) Could there be an old registry setting that still holds the old version string or something?
Nope, nothing with a registry. The version is checked on startup. I did however make a mistake on the server where an extra newline was making the uploader overreact :) it should be working without errors/notifications now. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.21 19:01:00 -
[97]
Edited by: Callean Drevus on 21/08/2011 19:08:12
Originally by: Salam Horani Edited by: Salam Horani on 21/08/2011 16:26:02 hi callean,
i can't seem to get the market scanner to work. i'm on a mac so i can't use your uploader. this is what i get when i browse the "market scanner" page:
"Uploader has no idea where you are. You will be able to build manual groups, but you will not be able to start scanning the market."
so i don't see the start button anymore.
p.s. any plans to release a mac version of your uploader?
Hey Salam,
Have you given the website trust? It requires trust before it will be able to see your IGB headers to determine where you are. It is however stupid that the website is not asking for this in the first place. I'll add that :)
I would like to have a mac version of the uploader, but I would not like to release the source code (black box to deter market manipulation, you can't mess with it if you don't know how it works). There are probably ways of working around that, so a mac (and linux) uploader will undoubtedly be a reality one day. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.22 23:22:00 -
[98]
I have a strong need to get this thread up a bit again. But other than that, I've also got to announce I've added item browsing (something notoriously lacking from the start), a separate downloads section, and I've begun with tracking the daily changes of price information (every day at downtime). I need to build an interface for that though.
We've also gotten to the point where I think inferred trades are more or less correctly tracked, so I will soon be adding aggregated statistics for those too (allowing you to see how many orders were actually bought or sold, instead of having to guess from the amount of transactions).
Oh, and I've received the first feedback item that wasn't by me, so I feel completely obliged to implement it. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.24 01:54:00 -
[99]
Quote: Migrating the site to a bigger server soon(ish). Seems my development server cannot take the pressure anymore, you guys are just too enthausiastic about uploading
I'm going to take this as Callean throwing out the challenge to crash his new server with too much uploaded data. Challenge accepted :)
Oh and if my uploader could handle more than 2 eve clients dumping data at once, I'd help crash your servers faster :) |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.24 09:35:00 -
[100]
Originally by: Domi Etymology Oh and if my uploader could handle more than 2 eve clients dumping data at once, I'd help crash your servers faster :)
One of the reasons I haven't tried to fix this in the newest uploader version yet --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.25 23:26:00 -
[101]
As all of you who've already visited the website again know, we have successfully migrated to a new server, which will provide ample growing space for the site. I do hope the website didn't slow down significantly, since the new server is located in the US instead of the EU, but I haven't seen any noticeable time lag myself :)
I imagine people will go mad because the old page is not redirecting you to the new one automatically, but I'll fix that soon.
Besides all that, I haven't accomplished much in the past few days. Please blame SPAZ (Space Pirates And Zombies) which is a very interesting game. I can heartily recommend it.
--- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Vessper
Indicium Technologies Hephaestus Forge Alliance
|
Posted - 2011.08.26 23:17:00 -
[102]
Couldn't find a way of getting prices for every item in a region. All the APIs seem to be geared around single items?
EveHQ Character App |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.27 01:21:00 -
[103]
Originally by: Vessper Couldn't find a way of getting prices for every item in a region. All the APIs seem to be geared around single items?
Currently there is no way to do that, other than to query items in blocks of 25 (the info function allows for multiple ids). Might create an API function that just gives that information for every item in a region too.
Just felt it was prudent to limit the number of items in the basic information call, since they're currently sorted and selected in a somewhat inefficient way. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.27 14:14:00 -
[104]
Edited by: Callean Drevus on 27/08/2011 14:18:26
Originally by: Vessper Couldn't find a way of getting prices for every item in a region. All the APIs seem to be geared around single items?
Still, I've added it for your and anyone else's convenience. Be wary though, the result set might be rather large :)
EDIT: See API page for details. |
Domi Etymology
|
Posted - 2011.08.28 09:14:00 -
[105]
lol, I derped and realised I hadn't updated to the newest uploader, so the last few days since you moved the server none of my uploads have counted to marketeer. Well, at least marketdata will have been happy :P |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.28 10:28:00 -
[106]
Originally by: Domi Etymology lol, I derped and realised I hadn't updated to the newest uploader, so the last few days since you moved the server none of my uploads have counted to marketeer. Well, at least marketdata will have been happy :P
Actually, that means I've still got them ;) but less directly. Only they haven't counted towards your total :P --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Vessper
Indicium Technologies Hephaestus Forge Alliance
|
Posted - 2011.08.28 11:31:00 -
[107]
Originally by: Callean Drevus Edited by: Callean Drevus on 27/08/2011 14:18:26
Originally by: Vessper Couldn't find a way of getting prices for every item in a region. All the APIs seem to be geared around single items?
Still, I've added it for your and anyone else's convenience. Be wary though, the result set might be rather large :)
EDIT: See API page for details.
Thanks, much appreciated!
I'll have a play around sometime today and let you know how it goes EveHQ Character App |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.28 15:56:00 -
[108]
Originally by: Domi Etymology I'm finding an interesting situation with the uploading client. If I'm running to eve clients dumping data, it goes fine. However if I add a third client, the uploader seems to freeze up after a few seconds. It sort of looks like it can't handle the speed that items are entering into eve's cache, or something.
Anyone else noticed anything similar, or am I the only nut who tries market dumping data with 3 or more clients at once :P
Hey Domi,
I've just added a third account (yay!) and I've found that the uploader seems to performing perfectly even when uploading with triple clients. Have you experienced this issue with the latest uploader version as well? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.30 07:02:00 -
[109]
Did you leave it running for a while? When i tried it before, it would start off okay, then start to freeze up over time. Haven't tried running 3 clients recently though, I'll give it another shot this week sometime.
Any idea if the uploader's code is written to take advantage of multi-cores on the cpus? Or is it like 95% of the software out there and still only single core? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.30 09:26:00 -
[110]
Originally by: Domi Etymology Did you leave it running for a while? When i tried it before, it would start off okay, then start to freeze up over time. Haven't tried running 3 clients recently though, I'll give it another shot this week sometime.
Any idea if the uploader's code is written to take advantage of multi-cores on the cpus? Or is it like 95% of the software out there and still only single core?
I am not aware of any multi-core features :P considering the load involved in what the uploader is doing, it would not be useful at all though. I think the overhead is mostly in disk access and upload time
I've left it running for a few hours (intermittently stopped when I needed to type something in the client), and saw a few slowdowns in a certain region, but it always recovered itself. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
|
Turhan Bey
|
Posted - 2011.08.30 15:18:00 -
[111]
Any thoughts on the upcoming historical market data dumps and how they may be used/integrated?
http://www.eveonline.com/devblog.asp?a=blog&nbid=2357
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.30 17:45:00 -
[112]
Edited by: Callean Drevus on 30/08/2011 17:47:05
Originally by: Turhan Bey Any thoughts on the upcoming historical market data dumps and how they may be used/integrated?
http://www.eveonline.com/devblog.asp?a=blog&nbid=2357
Hmm, if CCP starts providing consistent historical dumps, then the history upload feature might stop to be necessary. That would be pretty awesome. Except for the insane amount of data that would have to be imported every time, so I'd have to find a way to cut the time series short (or just get CCP to provide it with customizable limits or something).
Other than that, this data does not seem to provide anything more that history provides, though the fact that you'll always have consistent and available data is friggin awesome ^^
UPDATE: Do not know if it is visible for you guys, but like my signature? :) Have wanted to do that ever since adding the transactions page. ---
|
Turhan Bey
EVE University Ivy League
|
Posted - 2011.08.31 15:26:00 -
[113]
Sig looks good. :)
And yeah, I also like the idea of an official historical record provided by CCP. If anything, it gives us data that cannot be intentionally corrupted. I'll leave the logistics of regular imports up to you. :)
|
Muscaat
EVE Markets
|
Posted - 2011.08.31 16:07:00 -
[114]
I've written a cross-platform market data uploader that currently sends data to EVE Central and EVE-MarketData, and I'd love to be able to add support for EVE Marketeer too. Is your upload format documented anywhere? ---------- EVE Markets - global price trends at your fingertips |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.31 16:37:00 -
[115]
Originally by: Muscaat I've written a cross-platform market data uploader that currently sends data to EVE Central and EVE-MarketData, and I'd love to be able to add support for EVE Marketeer too. Is your upload format documented anywhere?
The upload format (for now at least) is the same as for EMD. The endpoint is different though, being http://www.evemarketdata.com/api/upload. Point the EVE Marketdata code there, and if 1 is returned everything went correctly, a 0 means some field was empty. Thank you for including my website in the uploader. ---
|
Domi Etymology
|
Posted - 2011.09.01 09:29:00 -
[116]
Callean, the "my orders" page seems to have developed a bug, it's remembering all my old orders, rather than use current information from the api. |
Iani Chieve
|
Posted - 2011.09.02 20:05:00 -
[117]
I am aware of the problem. I am also out of town for the next few days, so I will not be able to work on EMK for that time (no interwebz). I'll fix it when I return :)
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.09.04 20:41:00 -
[118]
Oh, it seems I have posted with the wrong character :) now you know one of my alts :P
In any case, I'm back, and will fix soon :) ---
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.09.06 19:05:00 -
[119]
Quick link to provide easy transition from this forum to the new one: https://forums.eveonline.com/default.aspx?g=posts&t=6686&find=unread ---
|
|
|
|
Pages: 1 2 3 4 :: [one page] |