Pages: 1 [2] 3 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Almir Kadric
|
Posted - 2011.08.02 10:58:00 -
[31]
BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?
And does it provide the actual route between those systems?
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 11:08:00 -
[32]
Originally by: Almir Kadric BTW the SystemDistances table you have, does it contain routes greater than 15-20 jumps, lets say systems 52 jumps apart?
And does it provide the actual route between those systems?
AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.
Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 11:23:00 -
[33]
Edited by: Almir Kadric on 02/08/2011 11:28:43
Originally by: Callean Drevus AFAIK, it shows every route between all systems. I just blew up my database checking whether it got distances higher than 20 jumps, so I'd guess yes. It's a pretty hefty table with 27M rows though. That was the first one I tried to eliminate when I found out the query was slow.
So it shows all routes and not just the shortest? That's how I did my original caching, but I've changed it since to only do shortest. Also I changed mine to only cache high sec systems since realistically not many would actually trade through low or null sec. And it shows the actual route, checked it against actually in game Auto Pilot so far identical. Only down side is i recently did the caching for 10-15 jump routes and it took 4 hours =_= currently figuring out a way to optimize it further, otherwise ill have to whip out my super computer in a box to do the cache once and for all then just save it somewhere.
Originally by: Callean Drevus Also, I don't know if you saw it, since I edited my previous post, and you replied so fast, but I had to edit the order statement to do the calculation again. Otherwise it wouldn't show me anything higher than 999 per jump. Any ideas?
EDIT: Just read your previous post edit. I see. That's unusual. I tested it on mine again and it seems fine. Did you change anything? show me your full query and ill take a look.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 12:15:00 -
[34]
Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields
There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.02 13:38:00 -
[35]
Originally by: Callean Drevus Ok, never mind, I found out what was causing the problem. MySQL doesn't like to sort on formatted fields
LOL didn't know that
Originally by: Callean Drevus There is actually quite a big problem with not having cached queries for me, as the data is updated every minute (changing the table, thus making caching useless). I guess I'll have to find a way around that, possibly just storing the result of the trades query in a different table for an hour or so.
I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.
And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 14:13:00 -
[36]
Originally by: Almir Kadric I don't think that really changes the resultant time that much. Especially if you're adding data on the fly. The database engine should be normalizing keys for entered data as you add them. The only time it slows down on first run is when you rebuild the table.
And to my next point why is your data changing every minute or so? You should realistically only be updating data every hour, or rather so doing a few items at a time where it takes an hour to rebuild the entire set.
Well, I try to have my marketdata as realtime as possible, so when someone uploads something new, I want those orders in my database as fast as possible.
Of course, I'm not calculating the traderoutes again everytime I get new data, that would be database suicide :) I just don't think the MySQL query cache likes it when I constantly add new data to mess up it's calculations. But I have not yet put serious research into that.
On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.
I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.02 19:13:00 -
[37]
Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.
I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?
It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.
I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent... --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.03 02:34:00 -
[38]
Originally by: Callean Drevus On a positive note, even after adding in all the other joins, the retrieval is still quite fast at 15-30 seconds.
Glad to hear it ^_^
Originally by: Callean Drevus I was just reminded of my ability to export my own database. So if you still need it, you can get the complete solarsystemdistances table from http://www.serial-experiments.com/solarsystemdistances.7z
LOL thanks for this, ill check what it contains and decides what's better, the one i built or this one.
Originally by: Callean Drevus
Well... I don't know what to say. Today I was working on making the trading information a reality, when I found out that some orders which are shown do not actually exist.
I thought this was weird, and I thought I might be doing something wrong with the import, but everything seemed quite fine. After a bit of tinkering I found out that my orders were actually correctly inserted, only to be suddenly reverted to the wrong state after a little bit of time. How was this possible?
It seems eve marketdata exports nonexistent orders. They even do that after they've just received the RIGHT information. After all, I just sent it to them. What was happening, was that my uploader sent the data to my server, where it got processed, it also sent it to the eve marketdata server where it was (presumably) processed as well. After my own processing everything went perfectly well, but then my lovely server got to the point where it retrieved all new data from eve marketdata, and surely, overwrote all the correct data with the invalid data that eve marketdata was sending.
I've now made it so that all data I recently received cannot be overwritten by eve marketdata data, but it still leaves the question of whether there is more incorrect data being sent...
Hmm sounds like eve-marketdata is caching the information first then only allowing exports after a certain time period. For example even though they get data in real time. They probably have that in a table which is only for inserts. Then they have a separate table which is built every hour or what ever time period which is specifically for select statements, which makes sense since they can speed up those statements, not to mention if they run a first run SQL cache statement to have sql ready for fast reads and just hot-swapping the databases over when they're ready.
Best to send them an email to find out what they're up to ;p
Let me know if you find anything
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.03 20:14:00 -
[39]
For everyone who is interested. The market scanner of evemarketeer is now able to scan only items you enter manually, therefore scanning only the items that are of interest to you. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.04 00:17:00 -
[40]
With the default upload suggestions for the market scanner, could I suggest you take out the Tech 2 BPO's, since I'm 99.999999% sure they aren't sold on the market anymore ;) |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.04 09:56:00 -
[41]
It seems there IS a marketgroup for them though :P as that is what I am filtering by. I'll look into removing them ;) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Temuil
|
Posted - 2011.08.05 15:09:00 -
[42]
I've been using this for a few days and liking it already.
I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.05 20:48:00 -
[43]
Originally by: Temuil I've been using this for a few days and liking it already.
I have a request if possible, when you search for an item and it shows the Regional Prices, can Molden Heath be added?
Jup, that is a legacy part of the website from when I was only using it myself. The other regions have been added/will soon appear. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.05 21:01:00 -
[44]
I almost forget that I have added a few new features.
You can now look up traderoutes within and across a few empire regions (these are updated every hour, you can see when they will be updated, so if you check a few minutes after that, they should be fresh). Provided data is actually uploaded in the regions where you try to find trades :) Trades older than 24 hours cannot be found, as it makes no sense, they'll be fulfilled anyway (I think, please contradict me if I'm wrong).
For now, eve central orders are not taken into account when calculating trades, since any and all orders I got during testing were either fake or outdated (even if updated 5 minutes previously). Didn't seem like a good idea.
You can also create a list of items to check their total value (and export and save the list to be used again later). Eventually saved on server, for now, you'll have to create your own text file :) I used this one to check the total value of items in a contract. Until CCP releases the Contracts API. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.07 19:58:00 -
[45]
I just updated evemarketeer with a feature that I think traders will love. It already exists on eve-marketdata of course, but I've streamlined the process a bit.
It involves checking whether your orders are really the lowest in the region. All updating of data is done via ajax, so the only thing you have to do is periodically check whether all the rows are still green.
Check it out at http://www.evemarketeer.com/trade/orders --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.07 23:04:00 -
[46]
In addition, I have an interesting image here, of what happens when you get your data from 3 different sources that for some idiotic reason do not align.
I'm talking about my own data, eve-marketdata and eve-central. Does anyone see a pattern here: http://www.evemarketeer.com/graph/tiny_history/10000043/20413/2? --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Cleuseau Remos
|
Posted - 2011.08.08 15:55:00 -
[47]
Callean,
Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.
Example would be: http://www.evemarketeer.com/api/jita_info/34/xml
Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.
TIA!
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 00:38:00 -
[48]
Originally by: Cleuseau Remos Can you add line terminators to your API XML? I'm trying to access data with an xpath and it is not working. My guru friend told me Xpath needs it.
Example would be: http://www.evemarketeer.com/api/jita_info/34/xml
Shows up great in a browser and I'm sure its fine as XML, but my Xpath parser needs line terminators.
I couldn't leave this alone :P
Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success). --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Almir Kadric
|
Posted - 2011.08.09 02:28:00 -
[49]
Originally by: Callean Drevus
I couldn't leave this alone :P
Something still tells me line terminators were not the problem, though I would not be hesitant about adding them if it were really the case. I've been trying with google docs myself for a bit just now, I made a few tiny changes to how the data was presented, and it seems to work now (google caches the same URL for 2 hours, so you'll have to change your URL a bit (add/remove a type_id or something) to see any changes in success).
Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 09:29:00 -
[50]
Edited by: Callean Drevus on 09/08/2011 09:29:36
Originally by: Almir Kadric Aren't you using an inbuilt XML parser to crate the XML? If so it should automatically present the XML ready for xpath use. Not to mention its easier coding to add on the fly as you use objects and an export rather than string concatenation. Hence cleaner looking code.
Actualy, I'm building the XML myself, since there'a always only a set of rows to display it I really didn't think about using the php DOM or XML functions. There are very little ways in which XML can not be ready for xpath use if it's well formatted. And given the depth of my XML, it's pretty much impossible to make any mistake.
What I was talking about when I said 'changes to the way data is presented' I meant made sure that a 'content-type: text/xml' header is sent, which google docs seems to like :)
PS: It is however, not impossible that I'll fall back to PHP lib functions if the output gets any more complex :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.09 20:32:00 -
[51]
People, I'd like to present my awesome feedback and prioritization form, allowing you to make your wishes known more easily. Please use it! --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.10 01:09:00 -
[52]
Eve's randomly down at the moment, so can't test the feedback form in the game browser, but trying to use it in firefox gives me "javascript:$('#descr_5').toggle();" in the address bar and "[object Object]" on the web page. Did I derp? |
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.10 13:10:00 -
[53]
Nope, you didn't derp, I did. Seems firefox doesn't like me putting javascript function in the href anymore. In any case, it should be fixed now. It's really unhandy if there's a bug in the feedback form :) --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Domi Etymology
|
Posted - 2011.08.11 12:10:00 -
[54]
Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.
For little bugs like this, do you want us to post here, or on the feedback thing you made? |
Turhan Bey
|
Posted - 2011.08.11 15:20:00 -
[55]
- Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.
Error: EVE Marketdata upload key must be set before upload! Uploaded 38 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 35 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 34 in Sinq Laison Error: EVE Marketdata upload key must be set before upload! Uploaded 39 in Sinq Laison
- What exactly determines the amount of "credits" given in the Top Uploaders list?
Top Uploads, Turhan Bey: 174,723 Top First Uploads, Turhan Bey: 70,787 Top Credits, Turhan Bey: 1,601,930.60
Top Uploads, Temuil: 72,783 Top First Uploads, Temuil: 10,251 Top Credits, Temuil: 1,716,783.58
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.11 17:36:00 -
[56]
Edited by: Callean Drevus on 11/08/2011 17:48:33
Originally by: Domi Etymology Trying to search for prices on R.A.M.-_Weapon_Tech seems to be busted.
For little bugs like this, do you want us to post here, or on the feedback thing you made?
Both are fine, if you can put it on feedback I will not forget about if the amount of posting here will suddenly increase for one reason or another.
Originally by: Turhan Bey - Attempts to upload to EVE-MarketData even when checkbox is deselected. The best workaround I can find is to empty your EMD key, in which case it spits out an error for each item, but at least it doesn't hang while the EMD site is down.
- What exactly determines the amount of "credits" given in the Top Uploaders list?
I see, this probably happened when you turned it off while the program was working. It was not updating a required variable correctly, leaving it in the previous state until restart.
In regards to the way the credits are calculated. I is based on how much orders have changed since the last upload, and on how much time has elapsed between then and now. It also takes into account whether there have been previous uploads for that item (hence the first upload thing), because if I did not, you would get MASSIVE amounts of credits when uploading information for an item for the first time (actually, I think that would grant you infinite credits ). What I think is that Temuil must have uploaded data that was very much changed since before, in regions that were very much outdated. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.11 20:27:00 -
[57]
I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.
|
Callean Drevus
Caldari Icosahedron Crafts and Shipping Silent Infinity
|
Posted - 2011.08.11 20:38:00 -
[58]
Originally by: Turhan Bey I've restarted the uploader (several times) after deselecting the checkbox. Even tried it one more time just now. It still believes that it is in the selected state.
Is the checkbox itself in the correct state after restarting? I'll have this fixed in the next uploader version. --- "A fool flatters himself, a wise man flatters the fool."
Chief Developer of EVE Marketeer. |
Turhan Bey
|
Posted - 2011.08.11 21:59:00 -
[59]
Yep, the checkbox itself appears correct. But the uploader behaves as if it were still selected.
|
Turhan Bey
|
Posted - 2011.08.12 16:20:00 -
[60]
re: credits
I assume that you have an upper cap on the time for repeated submissions, probably along the lines of a few days or a week. Otherwise, remote locations that may be ignored for many months at a time would generate credits also at huge levels. I would recommend treating a first-time submission at this cap level, i.e. reporting an item for the first time is equivalent to being the first to report it in N weeks/months at whatever is your cap.
Otherwise, what you are essentially telling your users is that it is more valuable for them to not go out to new areas and scan them, but instead repeatedly scan existing areas. It is also not valuable to keep remote area prices up-to-date, and they instead should wait as long as possible before going back and submitting updated info. |
|
|
|
|
Pages: 1 [2] 3 4 :: one page |
First page | Previous page | Next page | Last page |