Pages: 1 2 [3] 4 5 6 7 8 9 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.20 17:19:00 -
[61] - Quote
key is a free text field but it was intended primarily to be used as a simple ID number or hash. It could also be used as public key and hash for private/public key system.
Make sure you have read the JSON spec (one page) as it's important to understand it. http://www.json.org/
As pointed out "name" and "key" but not "origin" is part of uploadKeys. I'm just guessing but it seems you aren't using any kind of JSON library which make generating it correctly much easier. In the actual JSON the spacing etc used in the examples don't exist that was done purely for easier understanding of the format for us poor humans Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Desmont McCallock
164
|
Posted - 2012.04.20 17:37:00 -
[62] - Quote
Dragonaire wrote:In the actual JSON the spacing etc used in the examples don't exist that was done purely for easier understanding of the format for us poor humans Yeah, I too realized that while I'm trying to code the correct format using .NET JSON serializer. Had to copy it to a notepad and add the spacing myself, in order to be sure that I had generated it in the correct format.
On another note, to all market sites owners, expect during the weekend testing of EVEMon's uploader. |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.20 17:39:00 -
[63] - Quote
Desmont McCallock wrote:
On another note, to all market sites owners, expect during the weekend testing of EVEMon's uploader.
Will you feed it to EMDR only or directly to the sites as well? Creator of Eve Addicts |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.20 17:42:00 -
[64] - Quote
Desmont McCallock wrote:[ On another note, to all market sites owners, expect during the weekend testing of EVEMon's uploader. If you're testing, do feel free to point it at EMDR too. You won't **** anyone off too bad if you toss something badly formed in just yet, since EMDR is also in testing.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Desmont McCallock
164
|
Posted - 2012.04.20 17:55:00 -
[65] - Quote
First of, EVEMon is going to support only the unified format. Therefore only the sites that support the format can participate in testing. After all, afaik only EMDR has a gateway for the UF.
Edit: In order to answer to what might had been Snarf's question, EVEMon uploader will support preset and manually entered endpoints. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 03:39:00 -
[66] - Quote
In the interest of reducing complexity, I've simplified the upload instructions for clients down to a single URL:
http://upload.eve-emdr.com/upload/
I'll auto-detect the format and handle all of that boring jazz for you. This was motivated by some of you silly people trying to upload EMK format to the Unified endpoint. My fault for not making this smarter/easier from the beginning, though :) EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.21 06:25:00 -
[67] - Quote
Have you managed to fix the split data issue? If so, i can enable the stream again. Creator of Eve Addicts |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 06:37:00 -
[68] - Quote
Snarf Aldes wrote:Have you managed to fix the split data issue? If so, i can enable the stream again. I'm not positive on what action you wanted me to take, but check it out. I did change how EMD is feeding in pretty substantially. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.21 07:07:00 -
[69] - Quote
Ilyk Halibut wrote:Snarf Aldes wrote:Have you managed to fix the split data issue? If so, i can enable the stream again. I'm not positive on what action you wanted me to take, but check it out. I did change how EMD is feeding in pretty substantially. You treat buy and sell orders of the same item as separate uploads, although they belong together. Now you have one rowset of typeID xxx in region yyy for sell orders, and a second one of the same item + region for buy orders.
I figure that is because that is the way you pull it of Eve Market Data's API, but that would not be the way it was uploaded to them, or how it should be distributed again.
It would be better if EMD could be persuaded to forward their uploads the moment they get them. There should be no need for you to pull data from any api, although I understand that you want to supply as much data to EMDR as possible. Creator of Eve Addicts |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.21 07:12:00 -
[70] - Quote
I am now forwarding all uploads from Eve Addicts to EMDR. Are you accepting/processing history as well? I didn't receive any history uploads posted through Eve Addicts back via EMDR (using UF). Creator of Eve Addicts |
|
Desmont McCallock
164
|
Posted - 2012.04.21 08:23:00 -
[71] - Quote
Ilyk Halibut wrote:In the interest of reducing complexity, I've simplified the upload instructions for clients down to a single URL: http://upload.eve-emdr.com/upload/I'll auto-detect the format and handle all of that boring jazz for you. This was motivated by some of you silly people trying to upload EMK format to the Unified endpoint. My fault for not making this smarter/easier from the beginning, though :) All others should follow your example. |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.21 10:02:00 -
[72] - Quote
Since Callean put the source of the unified uploader online, it should be possible to write a unified upload part for the uploader. I looked at the code, but i'm a noob with python...
Anyway, here is a link to the code uploader source
If any of you python experts have the time to look at it... Creator of Eve Addicts |
Desmont McCallock
164
|
Posted - 2012.04.21 11:41:00 -
[73] - Quote
So according to EMDR documentation which is the correct upload gateway url, cause docs also mentions http://master.eve-emdr.com/upload/ (note the 'master)? |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 16:32:00 -
[74] - Quote
Whoops, I changed one reference, but not the other. This has been corrected to show the upload.eve-emdr.com hostname. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 16:34:00 -
[75] - Quote
Snarf Aldes wrote:I am now forwarding all uploads from Eve Addicts to EMDR. Are you accepting/processing history as well? I didn't receive any history uploads posted through Eve Addicts back via EMDR (using UF). History uploads are a bit less common, but I think we only get them from EVE Marketdata and direct uploads. There are not always history uploads to report, so they will be somewhat far and few between for a while. I'll be adding EVE Marketeer this weekend, which may improve volume somewhat.
If you make a point of watching and still see no history uploads at all, let me know and I'll figure out what's wrong. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 16:36:00 -
[76] - Quote
Snarf Aldes wrote: You treat buy and sell orders of the same item as separate uploads, although they belong together. Now you have one rowset of typeID xxx in region yyy for sell orders, and a second one of the same item + region for buy orders.
I figure that is because that is the way you pull it of Eve Market Data's API, but that would not be the way it was uploaded to them, or how it should be distributed again.
The data is grouped in EMD, but I thought I had corrected this. Apparently not, so I'll go back to the drawing board and come up with something.
Snarf Aldes wrote: It would be better if EMD could be persuaded to forward their uploads the moment they get them. There should be no need for you to pull data from any api, although I understand that you want to supply as much data to EMDR as possible.
It would indeed be a much better way to do things. I've touched bases with all of the major market sites, and have received responses ranging from "interesting" to "we'll get this done eventually". If you know any of them personally, perhaps share the fact that you're interested in such a thing to help them see the demand for this kind of thing. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.21 23:44:00 -
[77] - Quote
I'm pondering an architectural change, wanted some second opinions.
Currently, this is how things work, assuming we have two mirrored sets of daemons on two different machines:
http://www.eve-emdr.com/en/latest/overview.html#high-availability-through-shared-burden
This is working wonderfully at scale, we've got very high capacity, and processor usage barely peaks 3% on a low end Linode VM. The problem is, the gateway is a dumb relay. It does no parsing of the payload, aside from making sure a few relevant bits are there.
I'm noticing some mal-formed uploads, which get filtered out at the processor level. The problem is, the uploader has no idea that their data is bad. So what I'm considering is this:
- Get rid of the broker and processor daemons.
- Gateways take on the processing duties, and give the uploader feedback if their data sucks.
- Gateways send (PUB) the data to all Announcers (SUB).
Benefits of this approach:
- Getting constructive feedback to the users about bad data is possible this way.
- We remove some moving pieces.
- Less code.
- Less moving parts.
The downsides to this approach:
- Gateways will lose some throughput, since they'll be doing more.
- I'll need to look for gateway volunteers, instead of processor volunteers. This is worrisome only because the gateways sit behind round-robin DNS. If one of the gateways goes down and we don't automatically remove them from the rotation, a percentage of uploads will hang or never arrive. We'd need to write something to status check the gateways, and remove them from the round-robin DNS rotation if they become unavailable.
I'm leaning towards making the jump and doing this. Does anyone have anything to add? EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.22 00:27:00 -
[78] - Quote
The lack of timezone offsets has been corrected. All datetimes previously looked like this:
2012-04-21T20:25:55
They now look like this:
2012-04-21T20:25:55+00:00 EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.22 04:44:00 -
[79] - Quote
Today's round of changes
- You may now gzip-compress Unified Uploader Interchange POST bodies, as long as you set the 'Content-Encoding: gzip' header. This saves a ton of bandwidth, so do it! For .NET users, you may run into some issues, but Desmont is working on a way to do this.
- I have eliminated processors and brokers from the stack. This simplifies things, and lets me give uploaders direct feedback about whether their data sucks or not. Previously, it was too hard to tell if you messed up or not
- I will now return HTTP 400 error codes if you pass in data that sucks, along with a hopefully-helpful error message. HTTP 500's may be seen in the rare case where your data is so malformed that the gateway freaks out and doesn't know what to do with it
- I went postal on the various date parsing/encoding issues, and finally got everything pretty consistent. We're now more permissive than we should be as far as what kind of date strings we can parse, I'm not sure whether that's good or not :
We are now receiving data from the following sources
- EVE Central SMTP feed (orders only
- EVE Addicts (all direct uploads
- EVE Marketdata (orders+history via their recently updated API
The following data sources are in the works
- EVE Marketeer (all direct uploads, Callahan says sometime this weekend
- EVEMon Uploader (we'll be in default uploader endpoint, early June
And finally, a note on erroneous uploaders. There are quite a few who are uploading stuff that we can't use due to it being improperly formatted. Please, everyone, check your re-uploaders or syndicaters and make sure that your log files don't show a ton of errors now. Here are a list of a few EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.22 10:42:00 -
[80] - Quote
Question.. Why are you reformatting all the incoming data? I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID. Here is an example split_json.txt The top is what i am sending, the bottom is what is returned through EMDR.
I really need it to be in the top format, i cannot use the bottom format Creator of Eve Addicts |
|
Miilla
Hulkageddon Orphanage
234
|
Posted - 2012.04.22 11:13:00 -
[81] - Quote
So it gets its source market data from "bot" account toons that sit in every region and export the market data to your relay?
How is that not Botting?
CCP Don't give out real time market data to the REST API. So using BOT accounts is the only way to do it, in "near" real time, or depend on current real players to upload data, then you cannot trust that data.
If you are dependant on your data from external sites, that depend on players uploads, (or their bot accounts) then you are using second hand data again. It brings up the photocopy of a photocopy problem, data accuracy deterioration. HOw can we trust your data?
All you are doing is aggregating DIRTY INACCURATE market data.
The only real way to get anything "near" real time is to use account bot toons in every region and export every few minutes (or every 2 market refresh - automated - clicks) to your aggregator.
If you are not botting, you are pushing out second hand inaccurate dirty data as "fresh".
Another issue is, do you have an aggregation (page/data scraping) agreement with the other market data sites out there? What happens when they block you for scraping their pages? After all you are removing their eyeballs from their page, pushing up their hosting bill by more traffic and generally being a prick with them without an agreement. Even worse, If i was one of those sources you are scraping your data from, I would POLLUTE / CONTAMINATE the data just for your scraping bot. How can we TRUST your data isn't DIRTY by the source? This happens very often to hide from webspiders, in fact we can just keep generating random data for you TO INFINITY and keep your agregator busy, with such a tiny little script. Think tarpit :)
There should only be one source, from the CCP market, not third parties. |
Desmont McCallock
164
|
Posted - 2012.04.22 11:57:00 -
[82] - Quote
Snarf Aldes wrote:Question.. Why are you reformatting all the incoming data? I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID. Here is an example split_json.txtThe top is what i am sending, the bottom is what is returned through EMDR. I really need it to be in the top format, i cannot use the bottom format "uploadKeys":[{"name":"Eve Addicts","key":"relay"}], should be "uploadKeys":[{"name":"EVE Market Data Relay","key":"EMDR"}], as uploadKeys state where you are uploading. |
Desmont McCallock
164
|
Posted - 2012.04.22 12:01:00 -
[83] - Quote
@Milla You are a known troller. Can you go troll somewhere else, please. |
Miilla
Hulkageddon Orphanage
234
|
Posted - 2012.04.22 12:02:00 -
[84] - Quote
Desmont McCallock wrote:@Milla You are a known troller. Can you go troll somewhere else, please.
I will do my bit to pollute his market data, if he allows uploads :)
It will help me move some items at high profit :) |
Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.22 12:24:00 -
[85] - Quote
Desmont McCallock wrote:Snarf Aldes wrote:Question.. Why are you reformatting all the incoming data? I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID. Here is an example split_json.txtThe top is what i am sending, the bottom is what is returned through EMDR. I really need it to be in the top format, i cannot use the bottom format "uploadKeys":[{"name":"Eve Addicts","key":"relay"}], should be "uploadKeys":[{"name":"EVE Market Data Relay","key":"EMDR"}], as uploadKeys state where you are uploading. We can agree to use it that way, that would solve the duplicate insertion problem :) But.. i don't think that is the intended purpose of uploadKeys. What the actual purpose of uploadKeys is is unclear to me. Creator of Eve Addicts |
Desmont McCallock
164
|
Posted - 2012.04.22 13:10:00 -
[86] - Quote
From spec docs:Quote: uploadKeys; array; An array of objects containing per endpoint upload keys Endpoint is the entity to where you are uploading.
For example, once EVEMon Market Unified Uploader go live I will have to obtain an upload key from you so you can identify it and give credits to it. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.22 18:50:00 -
[87] - Quote
Snarf Aldes wrote:Question.. Why are you reformatting all the incoming data? I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID. Here is an example split_json.txtThe top is what i am sending, the bottom is what is returned through EMDR. I really need it to be in the top format, i cannot use the bottom format Sorry about this, I hunted it down and corrected this just now. It won't be a problem anymore. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.22 18:54:00 -
[88] - Quote
Snarf Aldes wrote: We can agree to use it that way, that would solve the duplicate insertion problem :) But.. i don't think that is the intended purpose of uploadKeys. What the actual purpose of uploadKeys is is unclear to me.
Since EMDR has no notion of an upload key, I just made that up from my scraper processes. I'm not actually modifying data, there just is no such thing as an upload key when I'm pulling the data from EC/EMD's feeds/API, and figured it might be useful to someone to see it in that format.
We probably do need to keep experimenting with solutions to the upload loop problem. I don't think the discussion is closed yet, and I'm willing to go outside the spec if need be. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Wind Jammer
Molden Heath Software Company
13
|
Posted - 2012.04.22 22:07:00 -
[89] - Quote
Nice work Ilyk. Your site looks good, and the architecture seems well thought through.
It's probably not ideal for my needs because I need an aggregator. I don't want to have to calculate my own averages or identify or filter out manipulation attempts. And I'm happy collecting new data once a day. But good luck all the same.
[url]http://eveproductionmixer.appspot.com[/url]
|
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.22 23:45:00 -
[90] - Quote
Wind Jammer wrote:Nice work Ilyk. Your site looks good, and the architecture seems well thought through. It's probably not ideal for my needs because I need an aggregator. I don't want to have to calculate my own averages or identify or filter out manipulation attempts. And I'm happy collecting new data once a day. But good luck all the same. Thanks! I guess the good thing is that you'll still benefit from it indirectly, in that wherever you pull your averages from will have better data.
Best of luck with your project. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
|
|
|
|
Pages: 1 2 [3] 4 5 6 7 8 9 :: one page |
First page | Previous page | Next page | Last page |