Pages: 1 2 3 4 5 6 [7] 8 9 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
6
|
Posted - 2012.06.18 05:01:00 -
[181] - Quote
Tonight's updates:
- Adjustments to the GoLang relay to figure out what error is leading to memory leakage.
- Restored EMK uploading to the gateways, for now...
- Added some additional useful error messages for incorrectly formatted UUIF messages.
- Dead code removal.
- Set up some automated alarms to notify me if there's an outage.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
7
|
Posted - 2012.06.18 08:45:00 -
[182] - Quote
I'm seeing something weird with respect to dates. The generatedAt value of 99% of the data has an hour field of 00, like this: snippet of received data:
Quote:2012-06-18T00:00:20+00:00 2012-06-15T00:00:32+00:00 2012-06-17T00:00:39+00:00 2012-06-17T00:00:05+00:00 2012-06-18T00:00:37+00:00 2012-06-18T00:00:20+00:00 2012-06-16T00:00:56+00:00 2012-06-18T00:00:38+00:00 2012-06-18T00:00:28+00:00 2012-06-17T00:00:09+00:00 2012-06-16T00:00:48+00:00 2012-06-10T00:00:46+00:00 2012-06-18T00:00:49+00:00 2012-06-18T00:00:55+00:00 2012-06-17T00:00:39+00:00 2012-06-16T00:00:42+00:00 2012-06-18T00:00:13+00:00 2012-06-18T00:00:32+00:00 2012-06-16T00:00:57+00:00 2012-06-17T00:00:20+00:00 2012-06-17T00:00:56+00:00 2012-06-18T00:00:44+00:00 2012-06-18T00:00:49+00:00 2012-06-18T00:00:14+00:00 2012-06-18T00:00:28+00:00 2012-06-11T00:00:33+00:00 2012-06-17T00:00:35+00:00 2012-06-18T00:00:14+00:00 2012-06-18T00:00:47+00:00 2012-06-18T00:00:57+00:00 2012-06-18T00:00:54+00:00 2012-06-18T00:00:46+00:00 2012-06-18T00:00:51+00:00 2012-06-17T00:00:12+00:00 2012-06-17T00:00:30+00:00 2012-06-18T00:00:26+00:00 2012-06-18T00:00:32+00:00 2012-06-10T00:00:36+00:00 2012-06-18T00:00:29+00:00 2012-06-12T00:00:33+00:00 2012-06-14T00:00:26+00:00 2012-06-17T00:00:00+00:00 2012-06-17T00:00:10+00:00 2012-06-18T00:00:37+00:00 2012-06-18T00:00:14+00:00 2012-06-18T00:00:49+00:00 2012-06-14T00:00:20+00:00 2012-06-17T00:00:35+00:00
I don't know where the problem occurs, at EVEMon or at EMDR, but this cannot be right. Creator of Eve Addicts |
Desmont McCallock
186
|
Posted - 2012.06.18 08:55:00 -
[183] - Quote
If those are 'history' messages it's perfectly normal as Entity explained in https://forums.eveonline.com/default.aspx?g=posts&m=1265226#post1265226 |
Snarf Aldes
University of Caille Gallente Federation
7
|
Posted - 2012.06.18 10:12:00 -
[184] - Quote
Nope, they are all from orders.
edit: Even for history the generatedAt value should be variable, its not the actual data from the 'rows' field, those are all dates with time 00:00:00. Creator of Eve Addicts |
Desmont McCallock
186
|
Posted - 2012.06.18 11:21:00 -
[185] - Quote
I just checked with EVEMon and it turns out to be an EMDR bug. |
Otaci
Infinite Improbability Inc Mordus Angels
0
|
Posted - 2012.06.18 11:48:00 -
[186] - Quote
Just confirming the 00:00 bug.
Number of occurrences of a particular {Hrs,Mins} time in a sample of 2750987 market snapshots:
[{{0,0},2749524}, {{9,57},33}, {{10,38},33}, {{10,28},30}, {{10,37},30}, {{10,30},30}, {{10,31},29}, {{10,36},27}, {{10,39},27}, {{10,40},25},
|
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
6
|
Posted - 2012.06.18 14:03:00 -
[187] - Quote
Is this a new behavior, or has it been this way for a while? If anyone with historical records could comment with certainty, that'd be awesome. I'll start seeing if I can reproduce it, in the meantime. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.06.18 22:17:00 -
[188] - Quote
Ilyk Halibut wrote:Is this a new behavior, or has it been this way for a while? If anyone with historical records could comment with certainty, that'd be awesome. I'll start seeing if I can reproduce it, in the meantime.
This is the first 30 rows from my DB so a few days old since I expunged:
+-----------+---------------------+ | orderID | generatedAt | +-----------+---------------------+ | 468766023 | 2012-06-16 00:00:43 | | 468840171 | 2012-06-17 00:00:52 | | 547489123 | 2012-06-18 00:00:47 | | 571460595 | 2012-06-17 00:00:50 | | 574327332 | 2012-06-16 00:00:25 | | 584959892 | 2012-06-16 00:00:30 | | 602992867 | 2012-06-02 00:00:44 | | 608251359 | 2012-05-29 00:00:17 | | 625302900 | 2012-06-02 00:00:06 | | 628410159 | 2012-06-15 00:00:49 | | 632715417 | 2012-06-09 00:00:08 | | 685886645 | 2012-06-10 00:00:04 | | 693936276 | 2012-06-17 00:00:52 | | 693943346 | 2012-06-16 00:00:01 | | 701055227 | 2012-06-14 00:00:35 | | 718577482 | 2012-06-04 00:00:48 | | 742684127 | 2012-06-15 00:00:10 | | 742684363 | 2012-06-17 00:00:28 | | 742688256 | 2012-06-17 00:00:08 | | 765443947 | 2012-05-29 00:00:03 | | 785285268 | 2012-06-11 00:00:58 | | 795473049 | 2012-06-14 00:00:23 | | 809346419 | 2012-06-13 00:00:37 | | 809347982 | 2012-06-10 00:00:30 | | 810868383 | 2012-06-15 00:00:57 | | 816612483 | 2012-06-16 00:00:08 | | 822470501 | 2012-06-17 00:00:26 | | 822877900 | 2012-06-13 00:00:28 | | 823070102 | 2012-06-16 00:00:49 | | 823071375 | 2012-06-16 00:00:35 | +-----------+---------------------+ |
Desmont McCallock
186
|
Posted - 2012.06.19 10:13:00 -
[189] - Quote
Problem solved. EMDR back in business. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
6
|
Posted - 2012.06.20 02:38:00 -
[190] - Quote
A european relay has been added. If you're across the pond, make sure you use that.
http://www.eve-emdr.com/en/latest/access.html
p.s., we'd love to have some more relays! EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
|
Snarf Aldes
University of Caille Gallente Federation
7
|
Posted - 2012.06.26 08:37:00 -
[191] - Quote
Just wanted to say that since the generatedAt issue has been solved, everything has been running very smoothly. The EVEMon / EMDR combination keeps Eve Addicts supplied with very up to date data.
Thnx to both, it works really well. Creator of Eve Addicts |
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.06.26 13:29:00 -
[192] - Quote
I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
8
|
Posted - 2012.06.26 14:38:00 -
[193] - Quote
Snarf Aldes wrote:Just wanted to say that since the generatedAt issue has been solved, everything has been running very smoothly. The EVEMon / EMDR combination keeps Eve Addicts supplied with very up to date data.
Thnx to both, it works really well. Thanks, glad to hear it's working well. The community has provided all kinds of useful feedback, help, and relays, which has made things go really smoothly. I hope we continue to see people share ideas about how to use EMDR, and that we can all find new and exciting things to do with the data! EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
8
|
Posted - 2012.06.26 14:39:00 -
[194] - Quote
Barl Rathbone wrote:I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier. That would be really nice, though, CCP has repeatedly expressed their opposition to providing real-time market data. I'm not sure they'd do anything to help our cause like that, though, it would be nice :) EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
8
|
Posted - 2012.06.26 14:40:00 -
[195] - Quote
Those on the mailing list have already seen this a while back, but I thought I'd also mention here that we now have four relays on two continents. Europe and the US each have two:
http://www.eve-emdr.com/en/latest/access.html
For those who value reliability as a chief concern, make sure you're connecting to more than one relay, and de-duplicating the messages that come in. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Tseehn Marhn
Eve Engineering Finance Eve Engineering
0
|
Posted - 2012.06.30 21:25:00 -
[196] - Quote
Barl Rathbone wrote:I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.
I've been trying to think of a way to find completed orders as well. An idea I had was this.
Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled.
This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does. |
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.07.01 00:28:00 -
[197] - Quote
Tseehn Marhn wrote:Barl Rathbone wrote:I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier. I've been trying to think of a way to find completed orders as well. An idea I had was this. Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled. This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does.
Which is what I do - but it would be nicer if a message was generated that I could trust a bit more. :) And your data is only good as long as people check that item in that region with the uploader running. Not a problem for trade hubs but you end up with stale data at times in less frequented region/item combinations which can be slightly annoying. |
Desmont McCallock
187
|
Posted - 2012.07.01 08:11:00 -
[198] - Quote
Absence of order id can also mean that that order was canceled or expired. |
Tseehn Marhn
Eve Engineering Finance Eve Engineering
0
|
Posted - 2012.07.01 17:43:00 -
[199] - Quote
Barl Rathbone wrote:Tseehn Marhn wrote:Barl Rathbone wrote:I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier. I've been trying to think of a way to find completed orders as well. An idea I had was this. Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled. This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does. Which is what I do - but it would be nicer if a message was generated that I could trust a bit more. :) And your data is only good as long as people check that item in that region with the uploader running. Not a problem for trade hubs but you end up with stale data at times in less frequented region/item combinations which can be slightly annoying.
Ah, well, good to know it should work. Also good to know no order ID means dead order.
|
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.07.01 19:46:00 -
[200] - Quote
Tseehn Marhn wrote:
Ah, well, good to know it should work. Also good to know no order ID means dead order.
Mind you I wouldn't trust a single order message -- I aggregate orders over time and then process through so I have a better shot of getting a valid "hit" rather than invalid data. It is all user uploaded stuff so you have to take it all with a grain of salt. :) |
|
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.07.01 19:47:00 -
[201] - Quote
Desmont McCallock wrote:Absence of order id can also mean that that order was canceled or expired.
True - but the point is still valid - that it's not an active buy/sell order any longer. The reason is opaque but the net effect is the same. |
Ydnari
Estrale Frontiers Project Wildfire
124
|
Posted - 2012.07.02 17:41:00 -
[202] - Quote
EMDR plus EveMon's uploader is good stuff.
I used to parse exported market order files to keep prices up to date in a private industry calculator and order tracking site I wrote, now I have that connected to EMDR and just have to look at items on market in game for it to update prices a few seconds later.
And now having access to price history is icing on the cake
Thanks to all the developers involved. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
9
|
Posted - 2012.07.03 18:53:00 -
[203] - Quote
Ydnari wrote:EMDR plus EveMon's uploader is good stuff. I used to parse exported market order files to keep prices up to date in a private industry calculator and order tracking site I wrote, now I have that connected to EMDR and just have to look at items on market in game for it to update prices a few seconds later. And now having access to price history is icing on the cake Thanks to all the developers involved. We're glad you like it. Feel free to stop by #emdr on irc.coldfront.net anytime! EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Tseehn Marhn
Eve Engineering Finance Eve Engineering
0
|
Posted - 2012.07.03 19:34:00 -
[204] - Quote
Barl Rathbone wrote:Tseehn Marhn wrote:
Ah, well, good to know it should work. Also good to know no order ID means dead order.
Mind you I wouldn't trust a single order message -- I aggregate orders over time and then process through so I have a better shot of getting a valid "hit" rather than invalid data. It is all user uploaded stuff so you have to take it all with a grain of salt. :)
Yeah, considering this and other things, I'm just going to scrape data from the cache for now. It sucks I have to click through the market, but I don't feel like trying to find ways to filter out the spoofed orders right now. Eventually though, I want to combine local and EMDR data - and use local data to help verify the EMDR stuff where you can.
BTW, thanks Desmont for the cache parser DLL. Saved me weeks of trying to figure out the cache on my own. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
9
|
Posted - 2012.07.03 21:25:00 -
[205] - Quote
I think what was meant was that spoofing is obviously possible, but we don't see much evidence that it's a problem right now. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Barl Rathbone
NIGHTMARE FACTORY INDUSTRIES ROL.Citizens
0
|
Posted - 2012.07.03 21:35:00 -
[206] - Quote
Ilyk Halibut wrote:I think what was meant was that spoofing is obviously possible, but we don't see much evidence that it's a problem right now.
Correct. I haven't seen any real issues and to be honest, the volume of orders received if you identify the top and bottom oddballs tends to even out, just like any 95th percentile. |
Schereau Lasomere
University of Caille Gallente Federation
0
|
Posted - 2012.07.11 04:32:00 -
[207] - Quote
I'm trying to develop an EVE market data processor and I'm feeding it data from EMDR. A really great service! Big thanks to UUDIF's and EMDR's developers!
I've got a question for the UUDIF and EMDR developers:
I've read the EMDR and UUDIF specs as well as searched the related forum threads for info but I'm still confused about the currentTime and generatedAt timestamps.
You can read the reasons for my confusion below but basically I'm not sure which timestamp to use/trust for data freshness verification.
Some observations and suggestions:
currentTime: The UUDIF spec says that the time "may also be changed by any forwarders as well when the message is re-sent" which I interpret as the change being optional.
If the time is NOT changed by a forwarder (EMDR for example), then this time, as far as I can figure out, would be the generator's UTC/local time and so can't be used for verifying order freshness since all generators' clocks most likely are not synchronized.
Issue: The EMDR documentation does not (?) say whether EMDR does change currentTime or not. I'm guessing EMDR is NOT changing the time since I'm receiving messages out of order based on currentTime value (reading data from a single relay, the default European relay in Germany).
generatedAt: The UUDIF spec says this time is "The date/time...when the rowset was created". From where is this time taken/time according to whos clock?
- Search result generation time according to CCP's server time?
- UTC time, based on player's computer clock, when the search result is stored in the game cache?
- Other?
According to the UUDIF spec, this time is supposed to be used for freshness verification. If it's the UTC time, based on player's local time, when the search result is stored in cache, then this timestamp can't be used for freshness verification since most players' clocks aren't perfectly synchronized.
The UUDIF spech doesn't state where this time should be taken from and so I'm vorried that uploader developers may use different sources for this timestamp. This makes me unsure if I can trust this timestamp as an accurate freshness indicator.
Suggestion: Update the UUDIF spec so it enforces a common method for determining this time, ideally search result generation time according to CCP's server time, if possible.
|
Snarf Aldes
University of Caille Gallente Federation
7
|
Posted - 2012.07.11 07:20:00 -
[208] - Quote
generatedAt is the time when the eve client writes the data to the cache file. currentTime is the time that the cache file gets parsed and sent by the upload program (evemon etc.)
Times should all be in UTC. Creator of Eve Addicts |
Desmont McCallock
187
|
Posted - 2012.07.11 08:08:00 -
[209] - Quote
As Snarf said. All times should be UTC.
As the creator of EVECacheParser (a .NET library based on Entity's Reverence Python library) let me explain where 'generatedAt' comes from. Each cache file contains the time of its creation (EVE Time which is the same as UTC) in Windows filetime format. This filetime when parsed gives as the 'generatedAt' time.
'currentTime' is generated by the uploader app to state when the UUDIF was created. EMDR is indeed changing that (as a forwarder it's allowed to) and its not the same with the 'currentTime' EVEMon generates. Although in the majority of cases that time can be the same as the round trip time can be less than a sec.
Hope this helps. |
Muscaat
EVE Markets
12
|
Posted - 2012.07.11 11:40:00 -
[210] - Quote
Snarf Aldes wrote:Times should all be in UTC.
I disagree. As long as the timestamps have the correct timezone on them then there's no reason why uploaders should be forced to convert them to UTC themselves. See https://groups.google.com/d/msg/eve-emdr/oc6aMya4V1Y/5GsWbmh0l9QJ for the discussion triggered by me being in UTC+1 right now |
|
|
|
|
Pages: 1 2 3 4 5 6 [7] 8 9 :: one page |
First page | Previous page | Next page | Last page |