I see that you are currently using the EMK & EMD format for uploaded data. What about the unified format?

However, I'm not sure how useful such a service will be for real world applications, as you don't seem to provide any kind of query methods to access already broadcasted orders. I'm not sure what kind of applications you had in mind when creating this. But being able to pull orders from a certain point back in time would certainly help for apps that won't be online 24/7 and need to keep in sync.

I created the EVECacheParser library in order to use it to built in EVEMon an uploader (EVEHQ will use it too at some point). I intent to support only the unified uploading format so to kinda 'force' the market data sites devs to support that format.
As soon as I have the uploader coded and I'm ready to test, I'll drop you all a pm to notify you about it.

Edit: And of course you will have to create another URL as the endpoint for those using a unified format uploader.

Further more I intent to provide you with code in C# for the using page.

We've had a good deal of interest, and the network appears to be handling the subscribers without a hitch. The outbound compression of the data has kept us under 32 MB outbound a day, with our current "replicate EVE Central's feed" data set. Considering that EVE Central appears to be sending us about 320 MB a day, this is a pretty nice ratio.

We'll see how the usage data looks as we add more market data (we're talking to other market sites about getting involved), and may find that we don't need to get very fancy with our relays at all.

1. Are you doing any error detection for bogus submissions that are not valid?

2. Which linode location are you located at,

and why haven't you setup a alias for; tcp://relay-linode-atl-1.eve-emdr.com:8050

Vaerah, I think you may be missing the primary usage case for EMDR.

As far as a market-driven site is concerned, the feed is the exact same thing as having player upload clients pointed at your site. The big difference is that instead of an inefficient smattering of HTTP requests, EMDR feeds you a compressed, fast stream of market data. The actual contents are identical to what a client supporting Unified Uploader Interchange format would be sending to you directly.

You can't think of EMDR as a substitute for actually storing the data, it is only a firehose to feed your data store. If you need historical data, store the stream, or fill in your gaps from external services.

So think of it as a substitute for needing to have uploaders pointed specifically at you. Additionally, it's a central place that the players can point their uploaders at, where the greater community of market sites can get to it. The current reality is that people are forced to solicit their users to point uploaders at their site, and data is supplemented by EVE Central's SMTP feed. If we combined more of our data, we'll all be better off.

Vaerah, I think you may be missing the primary usage case for EMDR.

But if you start with zero history data, it'll take months before the stored data is of good and / or statistical use.

agree with Vaerah, without historical data this does not seem much different than being apart of the unified data uploader tool.

Specifically because it introduces a different point of failure where a developer (like me) does not necessarily need a feed as much as snapshot of transaction history for the market.

Also, users like statistics.
With emdr there is no way to identify users, so no way to count their uploads.
For my site there is just a list of the top uploaders, but for eve-marketeer there is a point system attached to the number of uploads that let you do extra stuff on the site.

No idea how to fix that... personally i don't care about stats, but i do catch myself looking at the ranking every now and then.

EMDR Client in C#. Please add in "Using Data from the EMDR network" page.

The idea to gather several feeds into one network is awesome.

1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data.

EMDR Client in VB. Please also add in "Using Data from the EMDR network" page.

I'll be trying to convince market sites to relay their uploads to EMDR. Right now, the project is very young, so it's somewhat of an uphill battle. I welcome all the help I can get convincing the players that be that it's worth their time.

Ideally, each market site would re-POST every unique upload that comes to them via a player to EMDR, or hook into the top end of the feed with ZeroMQ and do it in real-time. The first method is the simplest, the second one is the most efficient. I've been in contact with EVE Marketdata and EVE Marketeer, so let's hope they'll come around!

Do you have an example of how to accomplish it with ZMQ (in PHP if possible)?

Eve Addicts is now integrating the data recieved from EMDR.
I have an uploader ready, just need to tweak it a little, so soon I will be forwarding all direct uploads.

Just thought someone here might also be interested in some code I started on for the unified format stuff done in PHP. https://sourceforge.net/projects/evemarketfeed/

So, I bastardized a three.js example to make a WebGL powered EMDR upload activity map

http://map.eve-emdr.com

Systems will light up whenever market data is reported. Upload some data and watch your region flash, or just wait for the eve-central data to spam.

I am already converting to unified format, so that's not a problem.
I do however see some significant overhead with posting it to EMDR. If i can do it with ZMQ, that should be more efficient.

I had to disable the EMDR integration for now,i'm getting weird data, probably because I made a mistake somewhere with the parsing. I have to look into that.

EDIT:
I have looked at my code, but can't find anything wrong with it. I haven't saved the raw json i'm getting from emdr, so i can't look at that. I am seeing mixed up data, like someone tried to upload 2 rowsets in a single message, but instead of making 2 rowsets the rows of both items got merged.

Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.

EMDR Client in Java. Please also add in "Using Data from the EMDR network" page.

Don't expect me to write the client in 'Scala' too. I have enough with all these programming languages I already know. Although I'll try to write it in C++ but I'm not promising anything.

Someone is sending malformed json data into the system. Their data has merged the rows section of a bunch of items under the typeID of a single item.
I have put the data online: bad_json.txt
The first item, typeID 3396 ('Industrial Construction' skillbook) has only 20 sell order and 11 buy orders in The Forge at the moment, the json contains 456 orders total.

There's actually good news on this front. This was a bug that I introduced with a late-night coding session, rather than something malicious: https://github.com/gtaylor/EVE-Market-Data-Relay/commit/62b49b293062d14999cb97ecf3127afb7d80e8ff#L0L33

I need another set of eyes on the data to make sure this fix is good. I'll write proper unit tests soon, had to get this in quickly between tasks at work.

Sorry for the hassle. Thanks for noticing this and letting me know, I greatly appreciate the help in testing this out.

"uploadKeys": [
{
"origin": "EVE Market Data"
}
],
"generator": {
"version": "0.1",
"name": "EMDR EVE Market Data API Feeder"
},

"uploadKeys": [
{
"name": "EVE Market Data", "key" : "an id that you use internally"
}
],
"generator": {
"name": "EVE Market Data Relay",
"version": "the version of production code"
},

I can get that changed later tonight, though, wouldn't it be up to the site who generated the key to decide whether to use straight numeric, or alphanumeric (hash?) keys? I could see value in not restricting to numerals, though, this is more of a decision for the specification thread, I think, as the current spec say: "This can be a simple ID number, or some kind of message digest"

Yeah, I think I'm going to need to pull, queue, and re-combine data from their API after some more shuffling to get it mixed together nicely. As you noted, the orders are separated.

In the meantime, ordering by the generatedAt attribs may help, as that is when the data came to be. That may be a more reliable means than what order it came into your uploader.

I'll fix the uploadKeys, too.

The whole uploadKey section is pointless. You can't use it for a point system since you can't tell if you already counted that specific piece of data via an other route...
Like i said before, you need to know where the data was sent, so you can tell if you have already seen it or not.

In the actual JSON the spacing etc used in the examples don't exist that was done purely for easier understanding of the format for us poor humans

On another note, to all market sites owners, expect during the weekend testing of EVEMon's uploader.

[
On another note, to all market sites owners, expect during the weekend testing of EVEMon's uploader.

Have you managed to fix the split data issue? If so, i can enable the stream again.

I'm not positive on what action you wanted me to take, but check it out. I did change how EMD is feeding in pretty substantially.

In the interest of reducing complexity, I've simplified the upload instructions for clients down to a single URL:

http://upload.eve-emdr.com/upload/

I'll auto-detect the format and handle all of that boring jazz for you. This was motivated by some of you silly people trying to upload EMK format to the Unified endpoint. My fault for not making this smarter/easier from the beginning, though :)

So according to EMDR documentation which is the correct upload gateway url, cause docs also mentions http://master.eve-emdr.com/upload/ (note the 'master)?

I am now forwarding all uploads from Eve Addicts to EMDR.
Are you accepting/processing history as well? I didn't receive any history uploads posted through Eve Addicts back via EMDR (using UF).

You treat buy and sell orders of the same item as separate uploads, although they belong together.
Now you have one rowset of typeID xxx in region yyy for sell orders, and a second one of the same item + region for buy orders.

I figure that is because that is the way you pull it of Eve Market Data's API, but that would not be the way it was uploaded to them, or how it should be distributed again.

It would be better if EMD could be persuaded to forward their uploads the moment they get them. There should be no need for you to pull data from any api, although I understand that you want to supply as much data to EMDR as possible.

Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format

@Milla
You are a known troller. Can you go troll somewhere else, please.

"uploadKeys":[{"name":"Eve Addicts","key":"relay"}],
should be
"uploadKeys":[{"name":"EVE Market Data Relay","key":"EMDR"}],
as uploadKeys state where you are uploading.

uploadKeys; array; An array of objects containing per endpoint upload keys

Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format

We can agree to use it that way, that would solve the duplicate insertion problem :)
But.. i don't think that is the intended purpose of uploadKeys.
What the actual purpose of uploadKeys is is unclear to me.

Nice work Ilyk. Your site looks good, and the architecture seems well thought through.

It's probably not ideal for my needs because I need an aggregator. I don't want to have to calculate my own averages or identify or filter out manipulation attempts. And I'm happy collecting new data once a day. But good luck all the same.

Sorry about this, I hunted it down and corrected this just now. It won't be a problem anymore.

Cool, I switched the EMDR stream on again.
I am seeing duplicates for every entry now, but that could be because the same data is originating from different sources.

Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.

My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently.

Now last thing that needs to be fixed is when uploading empty orders messages.
I tried to setup EMDR locally (on Windows machine) but after a days headbanging I gave up. Didn't expected that setting up a Python client would be so hard. What really got on my nerves was the fact that I had to download, build, install all those external libraries a Python project is using and then finding out that it still doesn't run.

I'll definitely be in need of relays in the near future. Been dealing with moving into a new house unexpectedly, which slowed things down for a while, but I'm back in action now. Aiming to get one or two more issues worked through this week, then do a fial documentation push, then look to get some relays set up.

Rackspace or Linode are both just fine. It just can't be Atlanta+Linode, since that wouldn't give us any useful additional redundancy. Data usage from Atlanta to another Linode location may be free, though.

I'll be sending the call out for relays on the forums, and on the mailing list: https://groups.google.com/forum/#!forum/eve-emdr

Keep an eye on whichever one works best for you. The mailing list is super, super low volume, as I've mostly been doing things here. That'll probably be more for important announcements once we're up and running.

Actually it's zmq.green that's giving me the issues. It's not officially included in the distribution and I have to build, install it manually. Although I managed to do this at some time, still It through an exception on me with "Name _Socket not defined" in zmq\green\core.py

Unfortunately my friend, although it's in the repo, it's not included in the distributed msi or source zip files.
Never the less I managed to get the source code from the repo, build it, install it but I can't make the EMDR source code to see the package. Now I get "ImportError: No module named green". Every other requirement is installed and working.

I think they're approaching a release, so hopefully it won't be as painful for much longer.

This will eventually be posted to the unified upload interchange format thread when it's in better shape, but I'm breaking all of the market data structures, along with serializers and de-serializers, out into their own project. Initially, this will just be for the unified format, but it'll be maintained in a manner to make it easy to drop others in if something new and better comes along.

https://github.com/gtaylor/EVE-Market-Data-Structures

There's going to be some pretty heavy lifting going on with this over the week, so I wouldn't use it for anything yet. It doesn't support the absence of orders/history for a given item+region ID combo yet, but that will be addressed shortly.

Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.

Nice work Ilyk. Very nice work!

Excellent, let me know if you see anything wrong. I'd like to have the data as scrutinized as humanly possible so we can catch anything before reaching "1.0".

Yikes, that is probably a silly mistake in my EVE Central SMTP feeder. I've tweaked this, can you see if it's acting as expected for you now?

Just a heads up. EVEMon 1.7.0 which will contain the EMUU will go beta on the 26th. So get prepared.

Hello,

This is an update to remind you that scheduled maintenance will be occurring for the host your Linode resides on shortly. The time window for the start of the maintenance is:

Monday, May 21st 2012 @ 21:00 - 23:59 (9 PM - midnight) EDT
(Tuesday, May 22st, 2012 @ 01:00 - 04:00 UTC/GMT)

When the maintenance begins, your Linode will be gracefully powered down. We're expecting the maintenance to be completed in under 30 minutes, at which time your Linode will be queued to return to its previous state (booted or powered off).

There is no need to issue any jobs for your Linode as part of the maintenance. We will update you via this ticket once the maintenance has been completed.

We appreciate your patience and understanding as we work to maintain our infrastructure. Please let us know if you have any questions or concerns.

Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.

Went over without a hitch. Let me know if anyone's consumers didn't re-connect automatically, it should have all recovered by itself.

I have some stats on EMDR messages here: http://www.eve-markets.com/emdr/

My consumers connected back automatically with no problems.

I have some stats on EMDR messages here: http://www.eve-markets.com/emdr/

One thing I haven't understood, from the beginning in your code logic, is why are you decoding and then encoding again a UF message.

I see. So the EMDR is a stream of all the individual player uploads, collected from the various websites.

I see. So the EMDR is a stream of all the individual player uploads, collected from the various websites.

The market data being discussed in this thread are copies of what you see in-game when you look at the market for a particular item. They are the buy and sell orders for a particular item in a particular region.

An example of an EMDR message is here: http://www.eve-markets.com/msgrecvd/view/0000902f-a8a0-4958-b028-253ea6495663

Its for typeid 8585 (Medium S95a Partial Shield Transporter), in region 10000042 (at the bottom).

You can see the sell orders (7th row is false) and buy orders (7th row is true).

There are thousands of these messages per hour.

Great job, awesome work !

Anyway it's possible to get buy/sell orders of a specific items in a Region/Station (a sort of filter by itemID,regionID, etc ... ) ?

This has nothing to do with EMDR and EMDR should not do any filtering as it's just a relay service.
The reason you are seeing old data is due to EVEMon 1.7.0 activation. Upon activation EVEMon is sending any data it finds in the EVE client cache (and deletes the files once it's done). I thought about filtering data in EVEMon before sending, but ended up that that's not EVEMon's responsibility as an uploader.
So this leaves the responsibility with the receiver, meaning the market aggregator.

I also had another question about the time format. Is there any chance of switching to something more computer friendly - like a timestamp or something? I'm sure I can convert the ISO 8601, but I'm not looking forward to it. VB is impressing me less and less the more I learn about it.

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

Finally, I'd also like to volunteer for any coding you might need.

Also the price values. Sometimes they're coming through as ints, other times as floats. Is this on purpose, or is something on my end casting things funny?

https://forums.eveonline.com/default.aspx?g=posts&m=974200#post974200

I just tried to reproduce this and couldn't. Are you sure they're not being parsed by your JSON parser as ints or floats? The JSON strings look correct, from the ones I sampled. Can you send a complete JSON string with the integers instead of the floats?

Are you using the .NET native json parser?
http://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer.aspx#Y0
Btw, I'll just repeat, prices should be cast to decimal.

Has anyone done this using only javascript?

What if some 3rd party data consumer website did not need all this data sent at all, but just something like:

record {

item_ID,
region_ID,
minimum_sell,
date_time_minimum_sell_last_update,
maximum_bid,
date_time_maximum_bid_last_update

}

You are free to pick out whatever you need. EMDR sends data out in the unified format, which has everything you need in order to selectively pick from.

Since EMDR does not have past data I will not be able to pick older orders (still well valid today) and merge them with the new ones to form a decent order book.

Since EMDR does not have past data I will not be able to pick older orders (still well valid today) and merge them with the new ones to form a decent order book. Did I miss something?

2012-06-18T00:00:20+00:00
2012-06-15T00:00:32+00:00
2012-06-17T00:00:39+00:00
2012-06-17T00:00:05+00:00
2012-06-18T00:00:37+00:00
2012-06-18T00:00:20+00:00
2012-06-16T00:00:56+00:00
2012-06-18T00:00:38+00:00
2012-06-18T00:00:28+00:00
2012-06-17T00:00:09+00:00
2012-06-16T00:00:48+00:00
2012-06-10T00:00:46+00:00
2012-06-18T00:00:49+00:00
2012-06-18T00:00:55+00:00
2012-06-17T00:00:39+00:00
2012-06-16T00:00:42+00:00
2012-06-18T00:00:13+00:00
2012-06-18T00:00:32+00:00
2012-06-16T00:00:57+00:00
2012-06-17T00:00:20+00:00
2012-06-17T00:00:56+00:00
2012-06-18T00:00:44+00:00
2012-06-18T00:00:49+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:28+00:00
2012-06-11T00:00:33+00:00
2012-06-17T00:00:35+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:47+00:00
2012-06-18T00:00:57+00:00
2012-06-18T00:00:54+00:00
2012-06-18T00:00:46+00:00
2012-06-18T00:00:51+00:00
2012-06-17T00:00:12+00:00
2012-06-17T00:00:30+00:00
2012-06-18T00:00:26+00:00
2012-06-18T00:00:32+00:00
2012-06-10T00:00:36+00:00
2012-06-18T00:00:29+00:00
2012-06-12T00:00:33+00:00
2012-06-14T00:00:26+00:00
2012-06-17T00:00:00+00:00
2012-06-17T00:00:10+00:00
2012-06-18T00:00:37+00:00
2012-06-18T00:00:14+00:00
2012-06-18T00:00:49+00:00
2012-06-14T00:00:20+00:00
2012-06-17T00:00:35+00:00

If those are 'history' messages it's perfectly normal as Entity explained in https://forums.eveonline.com/default.aspx?g=posts&m=1265226#post1265226

Is this a new behavior, or has it been this way for a while? If anyone with historical records could comment with certainty, that'd be awesome. I'll start seeing if I can reproduce it, in the meantime.

Just wanted to say that since the generatedAt issue has been solved, everything has been running very smoothly.
The EVEMon / EMDR combination keeps Eve Addicts supplied with very up to date data.

Thnx to both, it works really well.

I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.

I would love, love, LOVE if CCP would generate a market message that hits our cache when volumeRemaining hits 0 on an order (ie, it completes). That'd make my life a heck of a lot easier.

I've been trying to think of a way to find completed orders as well. An idea I had was this.

Consider that the market order data is uploaded as lists of orders, and these lists are grouped by items in a region. If there is an order id in the database for some item in some region, and that order id is no longer appearing in the uploaded lists of orders for those items in that region, likely the order has been filled.

This assumes that every uploaded list of market data for an item in a region contains every order available in that region; I'm not 100% sure it does.

Which is what I do - but it would be nicer if a message was generated that I could trust a bit more. :) And your data is only good as long as people check that item in that region with the uploader running. Not a problem for trade hubs but you end up with stale data at times in less frequented region/item combinations which can be slightly annoying.

Ah, well, good to know it should work. Also good to know no order ID means dead order.

Absence of order id can also mean that that order was canceled or expired.

EMDR plus EveMon's uploader is good stuff.

I used to parse exported market order files to keep prices up to date in a private industry calculator and order tracking site I wrote, now I have that connected to EMDR and just have to look at items on market in game for it to update prices a few seconds later.

And now having access to price history is icing on the cake

Thanks to all the developers involved.

Mind you I wouldn't trust a single order message -- I aggregate orders over time and then process through so I have a better shot of getting a valid "hit" rather than invalid data. It is all user uploaded stuff so you have to take it all with a grain of salt. :)

I think what was meant was that spoofing is obviously possible, but we don't see much evidence that it's a problem right now.

Times should all be in UTC.

Because by using UTC you don't have to worry about DST.

I agree that things would be a lot simpler if everyone just used UTC

Edit: Btw, as we discussed (and I proved to you) in the Google group discussion, the time value has to be in UTC but you clearly have to state the TZ diff if you are using it.

The data/time in ISO 8601 format (example 2011-10-22T15:46:00+00:00)

And now for some hopefully more exciting discussions :)

EMDR is indeed changing [currentTime] (as a forwarder it's allowed to) and its not the same with the 'currentTime' EVEMon generates.