Pages: [1] 2 3 4 5 6 7 8 9 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
1
|
Posted - 2012.04.13 05:55:00 -
[1] - Quote
After many late nights of hackage and head->keyboard banging, EVE Market Data Relay has entered its public testing phase. This means that it's not the most lively, hopping data source, but it's ready to be poked at.
EVE Market Data Relay is meant to be a high-availability, highly-scalable, real-time market data aggregator feed. I'll be collecting data from every source I can (the various market sites, plus player uploaders pointed directly at EMDR), and relaying it to you in real-time over a distributed network. All data is parsed and re-formatted into Unified Uploader and Data Interchange Format for your ease of consumption.
The idea is to lower the barrier to entry for developers who are working on, or thinking about working on market data-driven sites or applications (this is not limited to being useful on just websites). Instead of having to write boring pollers/scrapers/listeners for the various market sites, and/or get players to point their uploaders at you, EMDR will take care of that, giving you a firehose of data to subscribe to. Over time, we can point our users' uploaders at EMDR, and consolidate data, without some of the past concerns about stability and availability (our only single point of failure is Amazon's DNS service).
The "Using Data from the EMDR network" page shows how easy it is to get started consuming the real-time feed in a number of different languages.
For the curious/nerdy sorts, here's a High Level Overview of the project, as well as a glimpse into its innards.
FAQ
- A centralized data source sounds neat, but what about reliability?
- EMDR is built in a way that eliminates all single points of failure (aside from Amazon's DNS servers). Here's a chart and a run-down showing a dual-stack model.
- What does it cost to use this service?
- Nothing. Additionally, if your application is in the same data center or service as one of our relays, you may not use any of your allotted bandwidth (depends on your provider).
- Why is this better than downloading X Market Service's daily data dumps?
- Depending on your usage case, it may not be better. EMDR is more useful to those who want timely updates to market data, and those who don't want to write software that pulls data from a number of different market sites. We'll also eventually try to get players to point their uploaders in our direction so you don't have to.
- Why not just use EVE Central's SMTP data feed?
- EVE Central's SMTP service is the closest equivalent to EMDR. However, EVE Central is just one data source. We'll be aggregating anything else we can get our hands on, and soliciting uploaders. Also, the technology EMDR is built on makes it super easy to get connected and consuming data. It's also wicked fast/efficient.
- Is the source code available for this project?
- Yes, and it's BSD licensed.
- I have some resources to volunteer. What do I do to help?
- Get in touch with me and let me know, and as we move through testing, I'll get you pieced into the network.
EVE Alerter - Rally your troops! EVE Mail me to get your own corp/alliance alerter http://evealerter.com |
Desmont McCallock
160
|
Posted - 2012.04.13 08:21:00 -
[2] - Quote
I see that you are currently using the EMK & EMD format for uploaded data. What about the unified format? |
Shellac Brookdale
RAZOR Alliance
5
|
Posted - 2012.04.13 09:07:00 -
[3] - Quote
Wow, this looks really ambitious. The technology stack is also sexy. I'd love to play around with ZMQ a bit. However, I'm not sure how useful such a service will be for real world applications, as you don't seem to provide any kind of query methods to access already broadcasted orders. I'm not sure what kind of applications you had in mind when creating this. But being able to pull orders from a certain point back in time would certainly help for apps that won't be online 24/7 and need to keep in sync.
Another approach would be a http feed based solution such as: http://code.google.com/p/pubsubhubbub/ Its based on HTTP and would allow more flexible access to orders by e.g. using URLs for different market categories or regions, access to archived data and caching and load balancing methods.
|
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
2
|
Posted - 2012.04.13 13:32:00 -
[4] - Quote
We support EMK/EMD and Unified Uploader format. If you're talking about the instructions here, I couldn't suggest using Unified Uploader format because, ironically, the EVE Unified Uploader doesn't support the format yet :)
Are there any other clients that do? I'd love to get testing with them. I can share details on how to upload.
Desmont McCallock wrote:I see that you are currently using the EMK & EMD format for uploaded data. What about the unified format?
EVE Alerter - Rally your troops! EVE Mail me to get your own corp/alliance alerter http://evealerter.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
2
|
Posted - 2012.04.13 13:39:00 -
[5] - Quote
Such a service would probably better be accomplished by someone who is subscribed to our firehose, who stored and offered the data up to others. Perhaps one of the other major market sites will get on board eventually. However, historical data is outside the scope of EMDR. I'm not being stubborn, EMDR is just going to do one thing and do it really well. If someone wants to take the data and make it available on their market site, I will gladly point to them in the docs as a source of historical data that at least partially came through EMDR.
Shellac Brookdale wrote: However, I'm not sure how useful such a service will be for real world applications, as you don't seem to provide any kind of query methods to access already broadcasted orders. I'm not sure what kind of applications you had in mind when creating this. But being able to pull orders from a certain point back in time would certainly help for apps that won't be online 24/7 and need to keep in sync.
I guess it does technically rule out a lot of desktop/mobile apps, since they aren't online 24/7, but my primary intended audience is those that want near real-time access to market data. For that particular usage case, EMDR is synonymous to getting uploader data over HTTP, but much more fast, reliable, and efficient. In other words, this is probably most appropriate for websites, or real-time trading applications in the case of a desktop/mobile app (with historical backfilling from EVE Central or EMK/EMD). EVE Alerter - Rally your troops! EVE Mail me to get your own corp/alliance alerter http://evealerter.com |
Desmont McCallock
160
|
Posted - 2012.04.13 13:43:00 -
[6] - Quote
I created the EVECacheParser library in order to use it to built in EVEMon an uploader (EVEHQ will use it too at some point). I intent to support only the unified uploading format so to kinda 'force' the market data sites devs to support that format. As soon as I have the uploader coded and I'm ready to test, I'll drop you all a pm to notify you about it. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
2
|
Posted - 2012.04.13 13:59:00 -
[7] - Quote
Excellent. I quite literally just updated the uploading documentation to include specifics for Unified Format uploading. This may be useful for your test purposes.
Desmont McCallock wrote:I created the EVECacheParser library in order to use it to built in EVEMon an uploader (EVEHQ will use it too at some point). I intent to support only the unified uploading format so to kinda 'force' the market data sites devs to support that format. As soon as I have the uploader coded and I'm ready to test, I'll drop you all a pm to notify you about it. Edit: And of course you will have to create another URL as the endpoint for those using a unified format uploader.
If you let me know when you support the Unified Format, I'll add you to our list on the instructions page as a client that supports it, and may go as far as screenshotting the config process for our users. EVE Alerter - Rally your troops! EVE Mail me to get your own corp/alliance alerter http://evealerter.com |
Desmont McCallock
160
|
Posted - 2012.04.13 14:04:00 -
[8] - Quote
Further more I intent to provide you with code in C# for the using page. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
2
|
Posted - 2012.04.13 14:16:00 -
[9] - Quote
That would be awesome. Thanks!
Desmont McCallock wrote:Further more I intent to provide you with code in C# for the using page.
EVE Alerter - Rally your troops! EVE Mail me to get your own corp/alliance alerter http://evealerter.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
3
|
Posted - 2012.04.15 04:35:00 -
[10] - Quote
We've had a good deal of interest, and the network appears to be handling the subscribers without a hitch. The outbound compression of the data has kept us under 32 MB outbound a day, with our current "replicate EVE Central's feed" data set. Considering that EVE Central appears to be sending us about 320 MB a day, this is a pretty nice ratio.
We'll see how the usage data looks as we add more market data (we're talking to other market sites about getting involved), and may find that we don't need to get very fancy with our relays at all. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
|
LifeHatesMe
SKULLDOGS RED.OverLord
4
|
Posted - 2012.04.15 19:14:00 -
[11] - Quote
Ilyk Halibut wrote:We've had a good deal of interest, and the network appears to be handling the subscribers without a hitch. The outbound compression of the data has kept us under 32 MB outbound a day, with our current "replicate EVE Central's feed" data set. Considering that EVE Central appears to be sending us about 320 MB a day, this is a pretty nice ratio.
We'll see how the usage data looks as we add more market data (we're talking to other market sites about getting involved), and may find that we don't need to get very fancy with our relays at all. Holy batman, you use the same service I use! +1 for Linode. I do have to ask two questions tho;
1. Are you doing any error detection for bogus submissions that are not valid? 2. Which linode location are you located at, and why haven't you setup a alias for; tcp://relay-linode-atl-1.eve-emdr.com:8050
Let me know, thanks! |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.15 21:57:00 -
[12] - Quote
LifeHatesMe wrote: 1. Are you doing any error detection for bogus submissions that are not valid?
At this point, it's pretty basic, but we do catch some obviously bogus submissions. We'll be improving this with time, though. Ideas are welcome.
LifeHatesMe wrote: 2. Which linode location are you located at,
We're still in initial test deploy mode, so just Atlanta right now. We'll be looking to add additional relays on other providers and other regions within Linode (and other providers) once I have the setup process streamlined a bit. However, we're finding resource util to be so low that there is very little downside to connecting and consuming the feed from other providers right now.
LifeHatesMe wrote: and why haven't you setup a alias for; tcp://relay-linode-atl-1.eve-emdr.com:8050
ZeroMQ has no notion of aliases. It needs a URI to something, whether it be a TCP socket, a local IPC socket, or etc. When you write an application that connects to EMDR, you are using your language's bindings for ZeroMQ, which is maintained by someone other than me. So I really have nothing to do with that, and there's no way to "alias" the URI, since EMDR requires no "EMDR client". Make sense? EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Vaerah Vahrokha
Vahrokh Consulting
574
|
Posted - 2012.04.15 22:44:00 -
[13] - Quote
The idea is very good, but in RL finance the pure real time feeds are a minority and mostly used for pure buy / sell trading platforms.
All the others (I'd say 95 percent of the existing platforms) have a mechanism to query for past data up to present or from time X to time Y.
Without past data storage many of the advanced trading tecniques cannot be used.
Example:
This whole thread.
What can be achieved with data feed + history:
Tritanium, on professional trading platform
Nitrogen Isotopes, on Microsoft Excel graph
Nitrogen Isotopes, with very clear price action
Technetium, on a semi-professional trading platform
There is data available from 2009 (already digested in various formats including industry OHLC, on EvEMarketeer.com) but also older data (EvE Central stored data).
For how EvE markets are structured and because of the lack of leverage, the minimum resolution you can really work with is daily prices.
Having a continuous stream is fantastic but without the ability to go back at least 2 years (even with just daily data) it really falls short. Auditing | Collateral holding and insurance | Consulting | PLEX for Good Charity
Twitter channel |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.16 00:22:00 -
[14] - Quote
Vaerah, I think you may be missing the primary usage case for EMDR.
As far as a market-driven site is concerned, the feed is the exact same thing as having player upload clients pointed at your site. The big difference is that instead of an inefficient smattering of HTTP requests, EMDR feeds you a compressed, fast stream of market data. The actual contents are identical to what a client supporting Unified Uploader Interchange format would be sending to you directly.
You can't think of EMDR as a substitute for actually storing the data, it is only a firehose to feed your data store. If you need historical data, store the stream, or fill in your gaps from external services.
So think of it as a substitute for needing to have uploaders pointed specifically at you. Additionally, it's a central place that the players can point their uploaders at, where the greater community of market sites can get to it. The current reality is that people are forced to solicit their users to point uploaders at their site, and data is supplemented by EVE Central's SMTP feed. If we combined more of our data, we'll all be better off. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
LifeHatesMe
SKULLDOGS RED.OverLord
4
|
Posted - 2012.04.16 02:44:00 -
[15] - Quote
Ilyk Halibut wrote:Vaerah, I think you may be missing the primary usage case for EMDR.
As far as a market-driven site is concerned, the feed is the exact same thing as having player upload clients pointed at your site. The big difference is that instead of an inefficient smattering of HTTP requests, EMDR feeds you a compressed, fast stream of market data. The actual contents are identical to what a client supporting Unified Uploader Interchange format would be sending to you directly.
You can't think of EMDR as a substitute for actually storing the data, it is only a firehose to feed your data store. If you need historical data, store the stream, or fill in your gaps from external services.
So think of it as a substitute for needing to have uploaders pointed specifically at you. Additionally, it's a central place that the players can point their uploaders at, where the greater community of market sites can get to it. The current reality is that people are forced to solicit their users to point uploaders at their site, and data is supplemented by EVE Central's SMTP feed. If we combined more of our data, we'll all be better off.
|
Vaerah Vahrokha
Vahrokh Consulting
576
|
Posted - 2012.04.16 08:41:00 -
[16] - Quote
Ilyk Halibut wrote:Vaerah, I think you may be missing the primary usage case for EMDR.
I don't miss it, I actually used a similar service with Infinity Futures (a futures broker).
What you miss is that your service at time = 0 has value only in the technical achievement point of view and its finance POV value will slowly increase as the connected 3rd party websites will store more and more of the data.
But if you start with zero history data, it'll take months before the stored data is of good and / or statistical use.
If you are just after the technology per se, then ok. Auditing | Collateral holding and insurance | Consulting | PLEX for Good Charity
Twitter channel |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.16 13:48:00 -
[17] - Quote
Vaerah Vahrokha wrote: But if you start with zero history data, it'll take months before the stored data is of good and / or statistical use.
Again, EMDR is a replacement for having uploaders pointed at your site directly. It is for market sites to get their users pointed at a central place in order to combine incoming data feeds. Instead of X player's uploads just going to EVE Central or EVE Marketeer, their uploads go to everyone. That is the point, not historical data.
If you need historical data and you're not storing any of your own, EMDR is not for you. There is an entry in the FAQ above that says as much. EMDR is more appropriate for the sites that store the feed. For sites that are just getting started out, I recommend connecting to EMDR, and backfilling data from EVE Central, EVE Marketeer, and EVE Market data as needed.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.16 13:53:00 -
[18] - Quote
LifeHatesMe wrote: agree with Vaerah, without historical data this does not seem much different than being apart of the unified data uploader tool.
That's just the thing, you should be thinking of it as a uploader tool :) That's the point! It's an uploader firehose that anyone can use, instead of just one site.
LifeHatesMe wrote: Specifically because it introduces a different point of failure where a developer (like me) does not necessarily need a feed as much as snapshot of transaction history for the market.
Since you are currently dependent on other sites for your data, it actually reduces failure points. EMDR is highly distributed, and has no single point of failure. In fact, you can connect to multiple Relay nodes to protect yourself against one of them going down. You have no such flexibility with any of the other real-time means of getting market data (EVE Central SMTP and having uploader clients pointed at you).
The best way to think of EMDR is as an upload tool that combines data from all over the place for you. If you don't want/need real-time data, or don't want to store said data, it is not for you. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
4
|
Posted - 2012.04.16 16:04:00 -
[19] - Quote
Hi, I have been playing around with emdr over the weekend, and I like the idea.
I do however have some questions/remarks.
In an ideal world, everyone would upload only to emdr, and not to any of the market data sites. The sites would all get their data from emdr and would not have to supply any data to emdr.
Unfortunately, we do not live in an ideal world, so the situation will be something like this:
Users will upload to (multiple) market data sites and emdr, and market data sites will supply data to emdr (like eve-central is doing now).
Now consider the following situation, a user is uploading to a total of 4 endpoints: eve-central, eve-marketeer, emdr and eve-addicts. Eve-central, eve-marketeer and eve addicts all push the uploads they are getting to emdr. Lets say they are smart and donGÇÖt push anything they get from emdr back into the relay system.
Eve-central, eve-marketeer and eve addicts all also subscribe to emdr.
Now the user uploads data for item X, I get it directly, and I get it an additional 3 times from emdr (once from the direct user upload to emdr and twice from the 2 other sites).
Currently there is no way for me to see that I have already received that data directly. So I have to accept and process all copies I get from emdr.
This can be resolved, but it requires a change in the json layout. If the uploading client software sends its destinations, for instance the urls itGÇÖs uploading to, then I as an endpoint can see if the data was already sent to me directly, and just discard it if it comes in again via emdr.
For example: GÇ£destinationsGÇ¥ : [ {GÇ£urlGÇ¥:GÇ¥http://eve.addicts.nl/api/upload.phpGÇ¥} {GÇ£urlGÇ¥:GÇ¥http://www.evemarketeer.com/api/uploadGÇ¥} {GÇ£urlGÇ¥:GÇ¥http://master.eve-emdr.com/upload/eve_marketeer/GÇ¥} {GÇ£urlGÇ¥:GÇ¥http://eve-central.com/datainput.py/inputdataGÇ¥} ]
Destinations should not be deleted, only added to by intermediate sites if they upload it somewhere else. If I would see that emdr is in the list, I donGÇÖt have to push it into the relay system, if itGÇÖs not I can decide to upload it after adding the url for emdr to the list.
This would ofcource only work for uploaders using the unified format.
So, the bottom line, I like the idea, but at the moment IGÇÖm a bit concerned about all the duplicate data that I will have to process if I scubscribe to the stream. Without a way to tell if you have already seen the data before, it will require a significant amount of processing to work through all the duplicates.
Snarf Creator of Eve Addicts |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.16 16:13:00 -
[20] - Quote
I think the destinations field makes a lot of sense. I'll have to dig up the thread for discussing the spec and point them here. I help out some with the Unified Uploader client, so I could submit a pull request for this if it sounds good to everyone.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
|
Snarf Aldes
University of Caille Gallente Federation
4
|
Posted - 2012.04.16 16:14:00 -
[21] - Quote
Also, users like statistics. With emdr there is no way to identify users, so no way to count their uploads. For my site there is just a list of the top uploaders, but for eve-marketeer there is a point system attached to the number of uploads that let you do extra stuff on the site.
No idea how to fix that... personally i don't care about stats, but i do catch myself looking at the ranking every now and then. Creator of Eve Addicts |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.16 19:08:00 -
[22] - Quote
Snarf Aldes wrote:Also, users like statistics. With emdr there is no way to identify users, so no way to count their uploads. For my site there is just a list of the top uploaders, but for eve-marketeer there is a point system attached to the number of uploads that let you do extra stuff on the site.
No idea how to fix that... personally i don't care about stats, but i do catch myself looking at the ranking every now and then.
EMDR preserves the whole Unified Uploader message. Unified Uploader Interchange format supports uploaderKeys, which are used for this exact purpose. It's a matter of the upload clients using it.
But we are prepared, and I wholeheartedly agree that we need to preserve and relay such information to you. I'd love to see a breakdown of which site has the most people providing data to EMDR, as well. I've got some preliminary tracking in, but we're currently only being constantly fed by EVE Central. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
4
|
Posted - 2012.04.16 19:48:00 -
[23] - Quote
How does eve-central upload data?
When i was looking what was coming out of emdr, i basically saw only the data that i was uploading to you myself. Until at some point i got a burst of some MB's in data. Since i was looking at it on stdout i decided to kill the script :)
Looks like they don't send it through as soon as they get it, but batch it and send it at a regular interval?
Creator of Eve Addicts |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.17 03:32:00 -
[24] - Quote
We're subscribed to their SMTP-based batch service. This is currently the only option for real-time data feeds out there, which is one of the reasons EMDR came into being. The emails with market data come from EVE Central every 1-3 minutes, depending on some variability on their side. On our end, we parse the email and feed it back into EMDR.
This is probably why you saw the huge burst, and only the stuff you were uploading. I'm in discussion with some other market sites to get them to relay their data as well, but these things take time, especially when we're talking about a new service.
For those with market-driven sites, I'd love to help make it easy for you to share data! Please EVE Mail me and I'll work with you on it. It's very simple, and we all win. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Desmont McCallock
164
|
Posted - 2012.04.17 22:34:00 -
[25] - Quote
EMDR Client in C#. Please add in "Using Data from the EMDR network" page. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.18 01:17:00 -
[26] - Quote
Much appreciated! This has been added. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Nomad I
University of Caille Gallente Federation
59
|
Posted - 2012.04.18 09:29:00 -
[27] - Quote
The idea to gather several feeds into one network is awesome.
1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data. |
Snarf Aldes
University of Caille Gallente Federation
5
|
Posted - 2012.04.18 15:33:00 -
[28] - Quote
Nevermind, i'm an idiot.... Typo's are not my friend.. Creator of Eve Addicts |
Desmont McCallock
164
|
Posted - 2012.04.18 15:45:00 -
[29] - Quote
EMDR Client in VB. Please also add in "Using Data from the EMDR network" page. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.18 15:50:00 -
[30] - Quote
Nomad I wrote:The idea to gather several feeds into one network is awesome.
1 question: Do you want to gather data for the network or will the markeddata sites push data into this network? The 2nd approach would be like a torrent network for data.
I'll be trying to convince market sites to relay their uploads to EMDR. Right now, the project is very young, so it's somewhat of an uphill battle. I welcome all the help I can get convincing the players that be that it's worth their time.
Ideally, each market site would re-POST every unique upload that comes to them via a player to EMDR, or hook into the top end of the feed with ZeroMQ and do it in real-time. The first method is the simplest, the second one is the most efficient. I've been in contact with EVE Marketdata and EVE Marketeer, so let's hope they'll come around! EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
|
|
|
|
Pages: [1] 2 3 4 5 6 7 8 9 :: one page |
First page | Previous page | Next page | Last page |