|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Corax. PURgE Alliance
13
|
Posted - 2011.10.10 18:08:00 -
[1] - Quote
I like XML formats myself but they have to much overhead and use to much bandwidth. Some thing like CSV might work and is very light weight but is really limited and doesn't support any form of meta-data which probably will be needed. I'd suggest using JSON as almost every programming or scripting language either has a library or something to work with it or you can make a simple one yourself if needed. It's also an open standard with little overhead that is fairly easy to understand but let's you do some complex things with it if you need to.
The only other thing would be coming up with a list for what data is require and what can be optional included and a standard set of names/labels for everything. The reason for allowing optional information is it makes sure everyone has something in place to handle newer stuff it doesn't understand plus allow each site to add some additional meta-data for their internal use if they feel it is really needed. It also allows the testing of new features and making it available to others before all of the other sites add it themselves.
To go with the above all the sites should somehow keep the full data including stuff they don't understand so if they do decide take advantage of it in the future they can simply rerun the old data to do so. This also would allow sites to relay the info to another site that might be able to use everything. I could see the relay feature as being very useful to do things like asking other sites for what you missed during an outage or planned down time. Killboards have had something like this for a while and there's no reason it can't be done for the marketing data as well. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
13
|
Posted - 2011.10.10 20:46:00 -
[2] - Quote
I agree everything should be kept to a minimum but I was thinking as great as having a common cache reader feeding data to the sites I was thinking it could be expanded to something that would also allow people to feed their API data into them as well anonymously. I can see many marketers liking a way to make people in all of Eve aware of what they have up to sell or are buying. I could see this helping to fill in the under scanned areas. Why have to wait until they or someone else scans the market in their area to make that data available?
I know it not hard now to just make a CAK to give to site to make it available to them but one thing the market provides is you don't know whom you're buying from or selling to until after you've done so but if you give them the CAK it is traceable plus there some addition data there beyond what you might still want to share with them. Making the purely public part of that data available just like they can see in game might be something people would be willing to do if they had a way to do so.
Just was a random idea I had I don't know if people would interested in this but I thought I'd throw it out there for people to chew on. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
13
|
Posted - 2011.10.11 02:36:00 -
[3] - Quote
Like any user supplied data even the API data could be faked from the client but like you said it would make more information available which is a good thing and can be used to help detect the bad data. One reason I made the suggestion is I know at least in the case on Yapeal it spends a lot of time doing nothing but waiting on the API servers or the database server but it doesn't use much of the outbound network bandwidth so if people were interested I could look at adding it as an option. I can also see it being something any other applications that work with the API data could add. If all local applications that have marketing functions could easily add API and/or cache anonymous data uploading it would make a lot more data available and then the biggest problem is for the sites to handle the load Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
15
|
Posted - 2011.10.22 23:40:00 -
[4] - Quote
So after looking at the format used by the above two sites I came up with an idea for how to layout the JSON version of the same thing. Here's a couple of example for everyone to look at:
Quote:{$site.root}api/upload?data= {"upload_type":"orders","region_id":1,"type_id":1,"upload_key":"abc", "developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 15:46:00", "log":[ [2288279261,"s",30002053,60005686,1299999.99,40,40.0,1,"2011-09-10 12:16:33",90,32767], [2322655203,"s",30002053,60005686,1299998.54,88,25.0,1,"2011-10-18 21:59:51",90,32767], [2052179646,"b",30002053,60005686,351001.11,100,39.0,1,"2011-10-09 08:38:12",90,32767] ] }
Quote:{$site.root}api/upload?data= {"upload_type":"history","region_id":1,"type_id":1,"upload_key":"abc", "developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 16:01:00", "log":[ ["2011-09-10 12:16:33",0.01,2.00,0.30,404,50], ["2011-10-18 21:59:51",0.06,7.00,0.80,909,1] ] }
The above have been formated for easier reading. To better understand the columns in "log" you'll probably need to have a look at page from EMK when it's back up or get it from repos and just open in as local file in browser. You can find it in tpl/api/display.html. I gave the examples as GET request but POST is much better. I made this just to get some comments and with a minimum of changes to see if anyone else had ideas how it might be improved on. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
15
|
Posted - 2011.10.23 00:48:00 -
[5] - Quote
Ok sorry for the double post but I think I made an improvement already myself by adding a little more meta-data to them. I also updated the formating to make it a little clearer but of course forums screw up most of it as usual
Quote:{ "upload_type":"orders","region_id":1,"type_id":1,"upload_key":"abc", "developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 15:46:00", "log":{ "header":["order_id","buysell","solar_system_id","station_id","price", "vol_entered","vol_remaining","min_volume","issue_datetime","duration", "range"], "rows":[ [2288279261,"s",30002053,60005686,1299999.99,40,40.0,1,"2011-09-10 12:16:33",90,32767], [2322655203,"s",30002053,60005686,1299998.54,88,25.0,1,"2011-10-18 21:59:51",90,32767], [2052179646,"b",30002053,60005686,351001.11,100,39.0,1,"2011-10-09 08:38:12",90,32767] ] } }
Quote:{ "upload_type":"history","region_id":1,"type_id":1,"upload_key":"abc", "developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 16:01:00", "log":{ "header":["history_date","low_price","high_price","avg_price","avg_price", "orders"], "rows":[ ["2011-09-10 12:16:33",0.01,2.00,0.30,404,50], ["2011-10-18 21:59:51",0.06,7.00,0.80,909,1] ] } }
A nice thing about this version is it's easier for humans to understand and you could possibly reorder the columns in the rows, and add or remove some of them should the need arise in the future. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
16
|
Posted - 2011.10.24 08:21:00 -
[6] - Quote
Thanks for the catch on typo I've updated post to correct. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
18
|
Posted - 2011.10.25 03:30:00 -
[7] - Quote
Yeah they've said no to doing that so many times they don't even bother anymore. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.29 05:25:00 -
[8] - Quote
I was going to suggest something like RSS with JSON payload but after looking into RSS a little more it seems it can't handle what is needed so maybe just straight forward RESTful web application type interface might be the way to go. The format I suggested back in post #36 might be usable directly or with a few changes. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.29 22:47:00 -
[9] - Quote
I've also looked at adding the same thing to Yapeal to grab market order information but some other things have had a higher priority lately (SkyRim) for me
The idea with developer_key was to have some kind of site key that others could check mostly just to have an idea where it came from to prevent loops. It could probably be better named. I thought about having some kind of two part encryption system but then we'd have to deal with the whole trust issue and I thought it would be better let each site decide how they wished to handle that part but I think there needs to be some way inside the meta data to track at least where it says it came from. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.30 16:29:00 -
[10] - Quote
@Kaladr Linky no worky for me. It just times out but even www.eve-central.com is being really slow right now too and is probably going to time out. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 07:07:00 -
[11] - Quote
Like your ideas but I think it would be a good idea to still do the data part with the extra header stuff from my improved version.
"log":{ "header":["order_id","buysell","solar_system_id","station_id","price", "vol_entered","vol_remaining","min_volume","issue_datetime","duration", "range"], "rows":[ [...] ]
It would make it more self documenting which would allow developers that weren't involved in this thread to still understand what everything was and also makes any future changes easier if needed to add additional columns etc. It would also let developers have more freedom if needed in the order of the columns should it be ever be required in their application. And then there is of course the most important reason it makes it easier for anyone (read me) when I'm trying to fix something that broke at 3am and I haven't had any sleep for 3 days so it's hard for me to remember which came first vol_entered or vol_remaining etc The extra overhead is minimal but adds a lot of value IMHO.
I do like the idea of maybe using UUIDs but how would it be better than just using a MD5 or SHA1 hash of the URL with parameters? Also which parameters were you thinking about using/including in it?
BTW thanks for fixing it so I could see the linky Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 23:27:00 -
[12] - Quote
I updated the wiki so everyone can take a look and see if they have any other changes on it. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 03:12:00 -
[13] - Quote
generated_by is a good addition don't know why I hadn't thought of it. I added a history example for us to work with as well.
Quote:I do like the idea of maybe using UUIDs but how would it be better than just using a MD5 or SHA1 hash of the URL with parameters? Also which parameters were you thinking about using/including in it? Still wondering about this. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:11:00 -
[14] - Quote
Quote:UPDATE: In addtion, if we are going to use that header row. I'd call it 'fields' or 'data_fields' instead of 'header' as it doesn't really have much to do with headers, and we might as well be semantically correct :) The reason I originally named it 'header' was I was viewing it like the header line of a CSV file but it could be named differently. Maybe 'field_names' or 'column_names' would be a better name.
The naming that really bothers me more is 'log'. I used that because the original uploader examples I looked at used that but IMHO naming it 'data', 'contents', or 'payload' would make more sense.
Quote:... That at least, is the great plus of UUID in my opinion. Maybe someone else had some other good reasons to use it. Actually UUID version 3 and 5 are MD5 or SHA1 of the URL so there isn't really much difference except for the formating that was why I asking the question since it would be possibly a extra step without really adding any extra value.
I've just been giving this a lot more thought and I think we might be going at this the wrong way. Instead of trying to figure out a common digest or checksum let's simple make sure to include enough metadata so that we know the receiver will have everything they would need to make one to insure loops don't happen and leave it up to the developers what they use. The network bandwidth and memory buffering differences is nil as long as it's less than a single TCP packet worth of data or at most two for locations with a lot of data rows.
Looking at it from the network bandwidth angle it might be a good idea to think a little more about how stuff is named but we don't want to go to far trying to make it smaller and end up make it human unfriendly either. I'll look at changing some of the names on the wiki and see what everyone thinks. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:29:00 -
[15] - Quote
Would bring it more in line with API XML which might have some benefits on developer understanding and code reuse maybe. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:36:00 -
[16] - Quote
Should we go the rest of the way and call it 'result' and use currentTime too? Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 17:01:00 -
[17] - Quote
Ok after looking at both the Eve cache export data and Eve API MarketOrder columns I revised the 'orders' JSON format to use the same names etc in the wiki. This should allow developers from both areas to use it with minimal amount of data conversion. I used the same order for the columns as given by cache export data.
In the 'history' I decided to use the same order of columns as shown in game. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 00:50:00 -
[18] - Quote
I'm really sorry about the triple post but I thought this was important enough and unrelated enough to my others to use one.
I'll direct you to a new forum thread on a new project those developers reading this thread might find interesting. https://forums.eveonline.com/default.aspx?g=posts&t=4169 Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 08:36:00 -
[19] - Quote
Sorry I miss edited it it should work now. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 10:07:00 -
[20] - Quote
Was it in hg? If so you should still have it in one of the other copies out there. I might even have an older one floating around somewhere Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 18:54:00 -
[21] - Quote
So been think on the JSON order feed stuff a little more and wondering if it really needs to include solarSystemID since that can be inferred from the stationID. The same could be said for the regionID but since it's only used once in the metaData it's less of an issue IMHO.
Anyone else have a feeling about this one way or the other?
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 19:53:00 -
[22] - Quote
Guess it comes down to how normalized your DB is but since orders always happen at stations and they don't move from solar system to solar system there's really no reason to be saving it to the DB and only need to use join on selects which should be being cached in most cases anyway. If you're trying to keep some common per region or solar system stats table up to date it should be ran at the end of the feed insert/update so it's only running once per feed not on each row which would be very slow. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 22:01:00 -
[23] - Quote
Main reason I changed columns to be a list was to have it match how it was done in the APIs but either way works fine. Having it as an array would be better in that it'll directly match up with rowset rows.
So after making myself look a bit foolish on another thread the next question I have is should 'currentTime' be formated as in the API with'-' and ':'s or like is done in log file names with '.' and nothing? The API has advantage of never having anyone having the same confusion I did but the other format would let it be extracted directly by cache readers from the file name. It's understandable why CCP didn't use them in file names because they aren't allowed on many OSes or have special means but it purely up to us on the feed. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 23:06:00 -
[24] - Quote
I do wish CCP had used that for the APIs it really would have made everything easier. It might make sense to do that here and just have everyone manage the conversion in the clients.
So there's another related question here about the rowset 'issueDate'. Should it also be converted so all the date-times are in the same format? I think it would be a good idea and barring that since we do have 'issueDate' in the same format as the APIs then 'currentTime' should be the same as it.
I'm up for everything being in ISO 8601 as it would let clients use their local time in 'currentTime' and the times from the API and cache are already in UTC so they can be converted with simple string appending. I'll update the wiki as I think it's a good way to go with this one. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 00:50:00 -
[25] - Quote
Quote:On another note, whoever edited the wiki, please check the first example :) And disable spell checking addons while doing so. (volEntered - volunteered) I don't think it was my edit that did that but I fixed it anyway.
JSON is really easy to understand really the spec for it is like one page. [link url=http://www.json.org/]http://www.json.org/[/link] Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 04:58:00 -
[26] - Quote
Quote:1. Data encoding. While there's little to no string data communicated in these requests, I think it's important to make a note that any string data SHOULD be encoded in UTF-8. From JSON docs: A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.
Quote:2. It needs a recommended algorithm to generate 'unique' field value. Something as "simple" as md5 of a certain string value (f.e., program signature concatenated with datetime of a generated request). But not leave it as "oh, just put something there". I not sure if I'm understand what you're talking about so please correct me if I'm wrong.
If you are talking about a way to prevent network loops it'll be up to each site to decide how they detect it but some pointers on what fields might be used could be added to the wiki. Since this will be used from upload clients to sites or site to site pulls there shouldn't be any need to include something beyond the existing metadata.
In the case of cache readers they already usually delete the cache files after sending so they don't try to resend the same data. For something based on the API you could use the cachedUntil time.
In the case of site to site transfers it'll be done as pulls from RESTful service where the 'client' well be asking for just the updates they want and will need to decide based on the data if they are going to update their internal data. So for example a request for Amarr Shuttle orders in Kor-Azor would be something like www.example.com/EveApiFeed/Kor-Azor/Amarr+Shuttle/. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 09:46:00 -
[27] - Quote
Ok I made a change to "updateKeys" which should allow people to use the "key" for several different things. Also tried to make it clearer for people what can and can't be used from the JSON message for the "key" here as well. It may still need some refining but at least now everyone that wants to add something 'unique' to the message to ID it can without interfering with anyone else's idea how to do the same thing Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 16:39:00 -
[28] - Quote
I take it you guys didn't like my example above I'm sure you are right and it should be done as "%20" so I edited it .
So besides my bad URL what does everyone think of the change I made? I didn't realize there was a problem with how the "uploadKeys" was done before until I was trying to document it and didn't have anything for the Name column in the table. I'd also just finish deleting that whole "unique" thing again and it hit me it might be possible to do something that could be used for either or both without it interfering with anyone else ideas of how these things could be solved. So now we have a "key" where anyone that cares to can either do something neat and useful or just shoot themselves in the foot instead and have minimal impact on anyone else.
A 'BAD' example: Use "key" to hold to complete encrypted message minus "uploadKeys" and "currentTime"
'GOOD' example: Use "key" to hold a simple registered clientID so they can get credit for uploading market order data.
'BETTER' example: Use "key" to hold clientID like in 'GOOD' but also a digest of it and the complete message from the 'BAD' example. "key" : "{clientID}|{digest}"
There are many other possibilities but that should give everyone some ideas about how it could be used or abused. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 00:00:00 -
[29] - Quote
First problem we had was coming up with a list of field that would be used to form the hash or if using UUID version 3 clients would have to know their URL which usually is hard to come up with with most providers plus since it's usually based on IP which is served to them by DHCP also worthless. What if their neighbor also plays Eve and gets the same IP but upload from a different region but the UUID would be the same etc problems. There is that plus many other problems try to come up with one that works for everyone. The methods that could solve these type of problems had problem in some parts of the world because that use to high of encryption as well and those are just the ones I've thought of. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 06:18:00 -
[30] - Quote
Ok sorry for the short reply earlier but was trying to do it in 5 minutes at the end of my lunch Let me go over in more detail why IMHO it's a waste of time and bandwidth to add anything like an ID or any of the other proposed ideas. I'm going to use as my example a SHA1 of the complete package except of coarse itself
So the client computes the SHA1 and adds it into the JSON message before sending it. First notice it has to make the message without itself, compute the SHA1, then add it to the message somewhere which is a couple of extra steps already but since it on the client the extra work isn't really a problem. Now it send the message to say EMK
The EMK server receives the message and now has to decide what to do with it. One thing it can do as you said is check the SHA1 and decide if it's a dup. This is a bad idea and I'll show why. Let's say the sender actually is into metagaming and has decided to start sending fake market information to sites like EMT to game the Eve market. He has a script that generates huge amounts of fake market data based on the actual ingame data but all the real data from everyone else keeps getting in his way so he decides to start compute the SHA1 of the real data but put his fake data in the message instead so when EMK receive the real data it ignores it as a dup instead. So after a little while you as the developer on EMK figure out what is happening and start computing the SHA1 on the data to detect the fake data. Let's look at what it takes to do that.
First you need to pull the SHA1 out of the message so it's not in your way when you compute your own from the message, then you compare that with the one you received to try to decided if this one is the fake.
Now the question I have is if you are going to have to figure the SHA1 yourself on the server anyway why would you want to waste EMK's incoming bandwidth on something your just going to end up doing anyway plus extra work splitting out the sent SHA1 and doing the comparisons etc?
I could go on to show other examples with all the other methods but in the end all of them fail because they are based on trusting the data received from the client that you have no control over.
Now that I've pointed out why none of the simple ideas work let's look at one that can but why it has issues as well.
Something that will let you be some what reassured that everything is on the up and up would be to use a full digital signing system where they use the site's public key and their personal private key etc but that requires that every site and uploader to get the keys from a trusted party and keep them secure etc. and usually cost some RL money for one or both parties and will require some additional overhead computing and in bandwidth as will. I don't think anyone is going to do this it just doesn't make sense to work that hard for a little ingame data that isn't even real.
So to summarize it. It's a waste of time having the client compute anything which doesn't decrease the server load and any ID etc can be fake so it is useless. In addition it wastes bandwidth and computing resources that can be better used for other things on the site server. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 15:37:00 -
[31] - Quote
Ok my point is you've already received the whole thing before you can even check the UUID and at that point it only takes a couple microseconds at worse to make a SHA1 yourself to check if it's a dup. It's going to take as long to receive the UUID as it does to calculate the SHA1 or a MD5 of the whole message with as small as the message is. Part of my point is if you really want to do something like that you can simple use it in 'uploadKeys' and it won't interfere with anyone else doing something else. Both way get you to the same point just one way let's the method used change where the other doesn't allow for changes without causing problem for everyone else if they don't want too or have a different idea how to do things kind of like what's going on now
Just to make it clear I have no problem with the idea of UUID or anything else but from the start no one has shown where the extra bandwidth used has any real benefits. People keep seeming to think in one way or another it'll make their life easier but in the end you'll still have to decide if it a dup or not and probably validate the data on top of that so you might as well just do it yourself in the server. For what you actually are wanting to do you might as well just be looking at whether it's the same regionID and typeID and done at about the same time. I guess it's just me but how I see it is the actual orderID and data is what's important here and you're going to have to look at them to figure it out if you really have something new or not and there nothing that says a client can't be sending the same random UUID or different one with the same data. Once again there's nothing you can trust from the client that will do what you want.
I know you've posted on other threads about having to find ways to filter out bad or fake market data etc so why would you be so ready to trust them not to do things to fake stuff with this and now trust them? Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 16:53:00 -
[32] - Quote
Ok so I guess what we need to get everyone's input on a couple things here and decided which way to go. I've done my best to point out the problems and failing I see in trying to add any kind of ID/hash etc to the JSON message but if that's what most people want I'll go along with it because have the common format for everyone to use is what's really the most important thing.
Question One Should we include some kind of ID like one of the UUID versions or a crc32, md5 or whatever?
Question Two If the answer to the above is yes what are we going to use?
- crc32 - simple to make and use and should have few problems with dups.
- md5 - Also simple to make and use and chances of dups on the small data sizes of most messages is going to be nil.
- sha1 - Takes a bit more to make and probably overkill for something this small.
- UUID version 3/5 - By itself isn't enough since it's basically the URL/hostname which for most clients is a problem because of getting their IP from DHCP and there's a high risk of more than one Eve player getting it and causing conflicts.
- UUID version 4 - Numbers totally random as long as client has a good source of truly random number generator but most have at best a relatively poor pseudo-random source instead.
- something else ?
Question three If using one of the hash/digest methods above what parts of the message should be used or should the whole message be used? Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.08 00:12:00 -
[33] - Quote
Quote:I'll repeat again, that I'm not interested in using this for any kind of validation, just in determining whether I am going to do anything with the upload at all. I do understand that but why I'm saying it won't really do what you think it will. You can have someone sending different data but the same ID, different IDs with the same data and both of the first with faked data. There's also the possibility of a third party doing a man in the middle attack and doing the same things. So if the ID has no dependable connection with the actual data then you're just wasting your time having it. You end up either ignoring good data or getting mostly bad data if someone decides to.
Easiest way to make it worthless is to simple start sending the same fake data and count up through the ID range while doing so and you fill everyone's database with lots of useless data that will make it impossible to use the good anymore. That's where at least to hashes have an advantage as when the received and local one don't match you know something is wrong.
You also stated you basically want something that can act like a primary key from a database but the only way you can end up with something that does that is to have a common list where there's a single point in charge of issuing them which we just don't have.
This problem is one that's been around since before there were even computers but the solutions to it haven't really changed which is you can't do it without a single point of control.
Another way to look at it is as a synchronization problem to insure you don't have different issuers issuing the same timestamp for different groups of data.
There's other ways to look at this problem as I've shown above but they all come down to you have to trust the person giving you the data is being truthful which we all know you can't trust anything coming from a unknown party on the Internet If you can trust you have to verify it and the thing that needs verified is the data which your method ends up ignoring. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.08 17:29:00 -
[34] - Quote
Quote:2. Aggregator pushing data to another aggregator. One thing here it won't be push it'll be pull so receiver is ask for what it wants to see.
The only place push is used is from the client-originator to the aggregator and like you said aggregator always wanting the data though it does need a way to deal with a stupid client that keeps sending it the same old data but they have to do that no matter what so still don't need anything in the message as only the aggregator can really decided if the client is being stupid by somehow comparing the data to what it already has received.
Quote:3. Aggregator feeding clients (such as mining buddy or griefwatch). Once again they would be pulling. It wasn't clear if you where thinking push or pull here but once again they asked for it with pull so just like the aggregator above they'll have to decided if it's new. There might be a possibility here to use if-modified-since header which would be nice and save transfer dups and I think the aggregators should implement it even if not every receiver will use it.
Note that if-modified-since will also work with 2 above as well since the plan was to use the same feeds for both. The aggregator may need to start tracking when stuff is updated but I'm thinking most of them are in some way already so shouldn't take any real changes to be able to respond correctly to if-modified-since headers.
Quote:Add: gzip'ing the data would be desirable. Do you know if there'd be any major issues if client set "Content-Transfer-Encoding: gzip" or the like, and pack the data accordingly? This is at the HTTP level and really out side the JSON message. Like always if the client and server both support it there's no reason it can't be used.
Just to make it clear the JSON message that is being sent will be the data using normal HTTP protocols though nothing says they have to use port 80 etc just that that's the easiest by far. This is all planned as a RESTful service so using normal POST, GET, PUT, etc.
Quote:JSON format you've devised so far allowing for submission of multiple data blocks in single push, which, to me, is a good thing. The only problem I see is that EVERY block would have sender signature. Should probably put it into a separate block prepending the data blocks? Or, put it into HTTP header... You're touching on something I've been thinking about if in a little different way. As it is now there is both stuff like 'regionID' and 'generator' in the metadata header together. That's fine and I basically was move what had been GET parameters into the message from the original system but it could be improved IMHO as well.
If there instead was a metadata header for the 'generator' part with 'resultType', 'version', 'uploadKeys', 'currentTime', and of course 'generator' and the 'regionID' and 'typeID' where moved down into the 'result' some where then inside the result you could have multiple blocks of data for say the same 'typeID' but different regions or the other way around or even more than one of each so clients could even upload everything from the cache in one message which would save a lot of bandwidth for the aggregators not having to receive each as a separate connection and not having a lot of extra overhead being told over and over again who the data was from.
IMHO this would greatly streamline the whole process. I don't know how much of a change it would require in the aggregators to work with multiple 'data sections' but I can see it really helping them with their network bandwidth usage with more uploaders that they may end up with having a unified uploader etc that we are working to have. I'll look at adding in a new section on the wiki my ideas how the new JSON would look so we can look at them side by side and see what people think. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.08 18:24:00 -
[35] - Quote
Yes I understand some of them now use push but I think pull would be much better as there's a lot less work for them since there's no need to track who needs what sent to them. Amost all the old services that have used push in the past on the Internet have moved to pull instead. RSS is a good example of a pull service that killed off several old push services that existed before because it simply works better. I see no problem with those that wish to continue their push stuff doing so but IMHO the new stuff should be pull.
I've put the purposed new format up for people to look at http://dev.eve-central.com/unifieduploader/start Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.09 00:06:00 -
[36] - Quote
I can't find the the link or remember the search that lead to it but saw where someone had done research into how well several different UUID generator were at being truly random with version 4 and they found they weren't that random really as thought because most were based on the pseudo-random number generator in the system which all had clustering issues so the UUIDs also ended up being cluster over a much smaller range than there should be. It was mostly because if errors in the implementations but across all systems they saw biases that shouldn't have been there so it's a much small problem set than one would think by the number of bits used.
Ok on the push vs pull with aggregator-to-aggregator my concern is if the client isn't ready or able to receive the update then it's lost to it because there's no way to ask for it again. after all they are hopefully receive data from their own uploaders so I just figured it would make more sense for them to request it when they know they have time to work with it. It's also much easier for the another sites to poll and see if the other site is down then trying to push something to someone else and keep having to ask did you get it? you don't know if they got it and can't or decided to ignore you so after a time maybe you try it again and maybe decided to give up after a couple tries. With pull if a client can't get the data they can decided themselves if and when they really want to try again. Both systems can work but I think if it was me designing the site I'd pay attention to people like Yahoo, Google, etc that killed off they push services in favor to pull because of the easy in scaling that can be done with pull vs push and other issues they had. One thing to keep in mind is it easy to proxy reads but you can't proxy writes so push is limited to how much data one server can send but pull is limited by only how many servers and proxies you can set up. But that's someone else's problem to deal with
The rest gets to wait until after work Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.09 05:37:00 -
[37] - Quote
Ok I'll take it back it looks like they have start the swing back to push again http://en.wikipedia.org/wiki/Push_technology But as you're finding it is a bit harder to implement. I can see it working for the site to site stuff but it doesn't work as well to end users where usually for security their firewalls block push services. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.11 06:48:00 -
[38] - Quote
So my question is are we going to go with the new proposed version or the original from the wiki? http://dev.eve-central.com/unifieduploader/start Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.14 06:52:00 -
[39] - Quote
Ok I've updated the wiki page descriptions to match the new format.
I do have one question though and that is on the 'generatedAt' date/time what it's used for as it's not totally clear the reason for it being there to me at least. Maybe whomever added it can explain it a little bit? Or we can at least define how it's going to be used and maybe change the name so it's use is more apparent without needed to find it in the wiki etc. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.15 17:29:00 -
[40] - Quote
Quote:I'll be back, I still have to write protocol explanation, which forum ate last time and I haven't had a time to repost. Sorry to hear it got eat I've had that happen a couple times too I'll look forward to reading it when you get back to it.
Quote: The "generatedAt" could be moved to HTTP Date or Last-Modified header. (If you/your server care to fill it.) For it's purpose, it could be used in the scenario II, where you compile data off hand and present it to clients as cached payload. But as I said, you can do it the straightforward way by utilising HTTP headers.
That was kind of my thought. That's in part why I decided to ask the question because I don't really see where having another date/time in the message is adding any value but is just using up extra bandwidth. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.16 09:03:00 -
[41] - Quote
Ok I now better understand where for the aggregators having data on when the rows were collected is important so I have no problem with 'generatedAt' I just wanted to see why it was needed.
Quote:I just noticed the duplication of the columns item though, wouldn't it be better to take that one out of the resultset? Duplicating that will probably happen way more often than duplication of the header content (if you wanted to send multiple result types in one message), though it will also allow for a lot of flexibily to what kind of messages you can put in. You are right that having 'columns' in each of the 'results' is very flexible but also a lot of overhead. The question is does anyone really need the much flexibility and if not how to fix it.
We could bump columns up directly under 'results' but then we'll need to add another container like 'rowset' to hold the array of rows. That would work but still leaves the option open to add or move the columns around but only at the message level which is probably all that would be needed.
Example: "results" : { "columns" : ["price","volRemaining","range","orderID","volEntered","minVolume","bid","issueDate","duration","stationID","solarSystemID"], "rowset" : [ { "generatedAt" : "2011-10-22T15:43:00+00:00", "regionID" : 10000065, "typeID" : 11134, "rows" : [ [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038], [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null], [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039] ] }, { "regionID" : null, "typeID" : 11135, "generatedAt" : "2011-10-22T15:42:00+00:00", "rows" : [ [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038], [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null], [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039] ] } ] }
There are some problems with my example but should we try something like that?
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.16 16:19:00 -
[42] - Quote
After a little sleep and thought I figured out what didn't seem right with the above example. 'columns' needs to be moved up into main part of the message not just up under 'results'. Also thought 'rowsets' would be a better name now. I made the changes on the wiki instead of trying to fight with the forums to show the changes Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
23
|
Posted - 2011.12.16 23:23:00 -
[43] - Quote
This structure does save a lot of overhead but if anyone can really show where the versatility of the other way would be needed we can look at it. The other thing is the 'resultType' would have to be moved into the rowsets as well to mix order and history together which no one seems to have asked for before.
I just didn't see anyone really needing to mix order and history in the same message anyway. To go along with that there's nothing stopping you from including more than one message in a response as long as both parties want to do that and actually that will probably happen in the case of the pushes since it makes sense to just make a single connection to the receiver and send all the data to them when you can. I even see a case for the uploaders to make a connection and keep it open and send the scan results without having to reconnect all the time.
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.08 00:05:00 -
[44] - Quote
Thought I'd bump this back up to the top again before it got lost. Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.08 07:54:00 -
[45] - Quote
Using tokuDb instead of normal MyISAM or InnoDB tables is a great idea if you are dealing with really large tables. http://www.tokutek.com/ I've done some testing even on fairly small data sets and it's much better there too.
I also prefer to use MariaDB over normal MySQL as it seems to work better and has less bugs in the main code. http://mariadb.org/ Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.09 18:21:00 -
[46] - Quote
Everyone I know of that uses Yapeal in large deployments runs into the same problems with network I/O (rarely), DB bandwidth which when you look into it in depth turns out to be Disk I/O problems 99.9% of the time. Most shared hosting has very poor disk I/O so lots of memory to handle surges is a must but at some point it has to go to disk and then your 3GHz 8 way server just became a 50Mbps server in most cases Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Dragonaire
Corax. The Big Dirty
35
|
Posted - 2012.03.12 01:25:00 -
[47] - Quote
Just thought it would be a good idea to bump this back to the top again Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
35
|
Posted - 2012.03.21 16:49:00 -
[48] - Quote
Desmont McCallock wrote:Just a quick question. Why is 'issuedDate' being parsed into DateTime format and not use the original *NIX timestamp (which is actually how the data have it)? Is this some kind of DB restriction? This was discussed some where earlier in the thread but I'm not finding it right now. The reason we choose the current format is it is easy to understand for people and computers. It's not OS dependent and most databases can work with it directly or it can be translate easily for those that don't. Most programming languages also can handle it with little trouble when need and more important since we're developing a 'standard' it makes much more sense to leverage other existing ones that have been well thought out, peer reviewed, and is well documented.
One thing to keep in mind is this is a data exchange format and not a data storage format and as originally stated we wanted it to be understandable even by a programmer that had never seen it before without having to read a bunch of docs somewhere. Much like JSON itself being able to describe it with a single page and just a couple of examples with descriptions was at least part of my original concept and I believe everyone else's. We also wanted something that was compact but not to the point that it becomes hard to understand or use. There was some early talk about using XML but it's just to bloated for something like this so that's where JSON won out.
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.03.21 17:24:00 -
[49] - Quote
One thing that has stalled this whole idea is none of the upload sites accepting the new format yet as far as I know which is also part of my reason for bumping this thread as a small hint to those developers they need to implement this like they all said they would I've seen lots of sites jumping on the universal uploader stuff but except if I missed the change it was still using the old format which has some know problems
I even started a project on Source Forge where people could start adding example code/libraries in different languages so everybody wasn't going end up reinventing the wheel https://sourceforge.net/p/evemarketfeed/home/Home/ but I haven't had anyone contact me about adding anything Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.03.23 23:05:00 -
[50] - Quote
Yes it should be up to date as there real has been any changes to it for last couple months. It also has everything needed by all the developers that gave input on it. It's probably safe to say this is at least the Beta or first RC version of it and until it's starts being used there's no known bugs at this point Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
|
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.03.27 17:03:00 -
[51] - Quote
Quote:I'd much prefer some example HTTP request/responses demonstrating use cases than any words describing the intent. I think you've missed the point as the actual way the message is sent from the data source to the data sink doesn't matter though the expected sink would be sites like EMK etc but since the messages are also expected to be use for data exchange between sites like it as well. EMK or any of the others can act as sources to each other to help fill in holes in their data from when they are down for maintenance etc. The idea is to do much as killboards do now where they exchange kills to get a more complete picture.
Quote:In the example there are cases where regionID and / or solarsystemID are 'null', why? In my opinion this should never happen, the uploading client should provide all necessary info. That is true if you are only getting your data from the cache but if you are getting your data from the Market APIs etc that information isn't available. There are plans it sounds like for EveMon to act as a data source and I have plans to also add sourcing option from Yapeal and both would be using the API data which only includes the stationID. The idea is to increase the amount of sources where data can come from so more people can help out without having to always run the current cache based tools like the unified uploader. Part of the problem with relying on people running uploaders is they tend to all cover the main hubs only but the API would provide data from any place people have market orders up as long as they use something like EveMon or provide a ApiKey to anything that use Yapeal for their Market API data. There is no reason that other API libraries couldn't add something similar.
I think both of you could benefit from reading the thread start on about page 4 where the formats were hashed out between the developers involved at the time. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.04 05:22:00 -
[52] - Quote
Snarf Aldes wrote:It seems to me that all DB maintainers would benefit greatly if the extra processing is done by the client, not the server. But asking every project that could act as an uploader to add everything needed to fill in the data where they don't need it for anything but to make it easier for those trying to run a web site on an underpowered server doesn't make sense either. If I had to add a large part of the SDD and make the ConquerableStationList API required to be on to fill in the holes in the SDD I know that personally as a developer I wouldn't even consider adding it to Yapeal. The web sites already need the SDD etc to display the information to there user for the numbers to names stuff where not all clients do so it make more sense to have the web server do the job.
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.16 16:52:00 -
[53] - Quote
Why not use the uploadKeys array and just add your serial number or whatever? It's there to be used however you want. If every upload site had something that was added to it by the sources you'd have the same information without having to change anything or optionally you can keep track internally where you received data from and add something to all of your forwards to detect loops. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.16 17:28:00 -
[54] - Quote
As Desmont McCallock pointed out it's all the same problem and your site is just going to have to do the duplicate detection. What I'm suggesting is a way to help find duplicates and keep from getting into forwarding loops in cases where you receive data from some site and also feed to it. Each site needs to make sure they track data source and not send the data back to the source but that's outside of what this standard is made to address. It's a data exchange format not a routing protocol.
Would it maybe good for all the sites to come up with a system to make routing tables between themselves probably but that's separate from the data format. I'd be more than happy to help develop something along this line if everyone is interested in it. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.16 21:41:00 -
[55] - Quote
You are correct that something like this came up before and it was agreed there was no way that couldn't be abused and there are better ways to handle it.
IMHO some duplicate data is not a bad thing since it should help in spotting anyone trying to push bogus data into the system. It also help when sites go down due to bug/maintenance etc in that when they come back up they should be able to request fills from the other sites. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.17 03:27:00 -
[56] - Quote
The developers at CCP have repeatedly said no to any kind of market data export to the point they generally won't even response any more when people bring it up so what we have now is as best we'll get. As for some kind of transactionID they are available to the person that makes the market order via the API so if all applications that have API information would add option to forward the market order API it could be added some how to the existing format. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
36
|
Posted - 2012.04.17 06:33:00 -
[57] - Quote
First you have to ask yourself why you think it's important you know who is sending you the data? If it's just for some kind of credit system what does it matter if someone else want to give one of your users some extra credit? If you are trying to use it for anything else it's a mistake as you'll never know for sure who is sending something to you especially if you allow normal HTTP connections and don't limit the connections to HTTPS which probably doesn't really add any value. If you were using something in your key that is used to authenticate the user that's just asking for trouble anyway and shouldn't have been done to start with. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.17 16:42:00 -
[58] - Quote
issueDate has to be checked or you risk overwriting new data with older data and the other checks I'm not so sure are needed. I believe if you change the price it gets a new orderID from what I remember from working with the marketOrder API. The issueDate would also change in that case which if I remember right is what happens in game when I've made changes to sell and buy orders. The volRemaining could be changed do to the low res of the date-time used where one scan sees a transaction and another doesn't but that should be a rare case and can be simply resolve by always using the smaller amount.
Quote:I would like all of your opinions about the 'destinations' info in the json format. You should be tracking where the data comes from internally and know not to send it back to where you got it. As posted above this is about route between sites which is outside of what the format is about which is a common format to exchange the data. There could be a separate route discovery process between sites that could use JSON as well but it shouldn't be a part of the data. You can always add your own uploadKey to the data to help detect loops but it'll be of limited use as pointed out above in the last couple of pages. As Desmont McCallock pointed out in https://forums.eveonline.com/default.aspx?g=posts&m=1140194#post1140194 it's really just a duplicate uploader problem that you have to deal with anyway and doesn't need to be handled differently. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.17 17:20:00 -
[59] - Quote
I thought it did on changes to price but I could be wrong on that or CCP made a change so it doesn't anymore without telling anyone but either way it can be handled simply by the receiver when processing the data. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.17 23:26:00 -
[60] - Quote
http://dev.eve-central.com/unifieduploader/start If you haven't read completely through that you should. uploadKey isn't made to really be used for what you think is needed but it can be used to detect message looping and all sites should have checks in place for them if they both send and receive data.
As pointed out by Desmont McCallock wither the duplicate data happens because you have 1000 users all doing market scans in say Jita or least people uploading to multiple sites which are cross feeding each other doesn't matter you still have to deal with dups in some way and truthfully both cases can be handled the same. I think part of the thought behind wanting some kind of tracking history like you are asking for is that you view the dups as a waste of resources vs where I and others see it as an chance to receive fills when my site might be down or confirming the data I've already receive from other source hasn't been tampered with. By the nature of how the data is provided you are always going to have to deal with dups so it's much better to design you system so it doesn't matter from the start then trying to have something added on that itself could just as easily be faked and worthless. I also don't see this being a big problem as there really isn't much need for more then 3-4 upload sites to begin with and them cross feeding each other.
On a relate thing I do think all upload sites should have some kind of uploaderID which they include in the uploadKeys so sites can try to be smart about which other sites they might forward to first. So for example if they notice an EMK key they might delay upload there until they've uploaded to Eve Central which isn't listed. I can even see having a system where by default you don't do pushes to other sites listed as receivers in the data but they can request the data even if they might already have it as a pull. Being able even to request filtered vs unfilter updates would probably be a very useful thing.
Anyway that my thoughts on making any changes. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
|
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.21 14:56:00 -
[61] - Quote
Desmont McCallock - You are correct that adding a 'z' for UTC is allowed even preferred but having no zone offset would mean the DT was in local zone which isn't useful in this application What you are probably running into is the server is in UTC so .NET doesn't think it needs to add it. I had to do this in PHP to make it do what I needed so you might need to do something similar gmdate('c', strtotime($row[$k] . '+00:00')); You could append just 'z' but I felt it would make it easier for anyone that has to parser it manually to always work with numbers.
Nothing jumps out at me as invalid accorded to the format in your example other than the DT issue.
As to empty order messages I guess it would depend on how the receiver site uses that info. If they take it to mean there aren't any orders but they show some that could be useful information but it'll depend on what each site decided which would be outside of what the format cover of course Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.21 17:00:00 -
[62] - Quote
I've added some more info to the format page to help make it clearer what is expected with the dates. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.24 05:04:00 -
[63] - Quote
Ilyk Halibut - You have some good points and it does need to be clearer what is a forward, generator, etc. I'll try to make a stab at it here and get some input from everyone.
Endpoint - Something that receives a message from either a generator and/or a forwarder and uses it in some way. A endpoint may also be a forwarder but is more commonly some kind of market research site, etc.
Forwarder - Something that forwards a message from a generator to something else and may only modify the "currentTime" and "uploadKeys" as stated in the existing docs.
Generator - Something that gets data directly/indirectly from Eve database either from cache files of Eve client or MarketOrder API and first sends the data as a message to an endpoint. Other things that would be considered a generator would be any endpoints that receive messages from generators and/or forwarders and then merges the messages or the data in the messages from the multiple sources together and then sent the combine data as a new message.
So let me know if that makes things clearer or if anyone have anything else to add to it etc. Everyone's input on this is welcome as we try to better define stuff. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.25 15:30:00 -
[64] - Quote
I'm not sure if the lack of comments on my last post is a good thing or not Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
39
|
Posted - 2012.04.25 16:01:00 -
[65] - Quote
Quote:Note here that if a reference to C is found in the GÇ£uploadKeysGÇ¥ field then there is no meaning in sending the UF message to C, as it has already been sent. That's assuming that C actually received it which might not have happened.
Quote:Special attention should be given when B decides to send the received UF message to C (acting as a forwarder). According to the UF specs, a B is allowed to modify only the GÇ£currentTimeGÇ¥ and GÇ£uploadKeysGÇ¥ fields. In the case of the GÇ£uploadKeysGÇ¥ field, B should erase all GÇÿkeyGÇÖ values as there is no point in forwarding them to C. They are irrelevant to anyone else. I can see cases where someone might want the uploadKeys for farther processing so just dropping them without reason is IMHO a bad idea. They should only be dropped in cases where either something has changed that makes them invalid or where it is unknown if they are still valid.
As to detecting duplicates it seem everyone is ignoring the simplest thing. The orderID and when it was generated and received. If you are receiving something from a source that matches existing data you need to only determine if its an update or outdated. You may also what to weight how trusted the source is etc but that's really out of scope of what UF's purpose is. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.04.26 14:21:00 -
[66] - Quote
I do understand the concern and when I get around to adding uploading to Yapeal there may be a few more sources showing up as well though EveMon probably will be the bigger change in numbers My main point is that UF was created to be a simple data exchange format not a routing and duplicate data detection system as how that is handled is up to the endpoint receiving the message. UploadKeys was added as an optional part that allows data sources and receiving endpoints to be able to pass a one way basic pre-agreed authorization or ID not to solve routing issues etc.
On the possible database load issue I have to ask a question here. Are you already seeing 1000s or 10000s of queries per second now? The reason I ask is I know with a well designed DB (correct indexes, and normalized) and well designed queries that a $15US a month single cloud node can handle that kind of load. If you are seeing less than a couple hundred per second and having problems now you need to be figuring out why. I know from my own experience in working on Yapeal that some very simple changes can make a huge difference especially with anything that does a lot of writes like Yapeal or anything trying to merge lots of data like the current endpoint have to. Hopefully no one that is using MySQL is trying to use MyISAM instead of InnoDB or TokuDB. The table locks and no transactions will kill you quickly. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.01 23:10:00 -
[67] - Quote
Packtu'sa - Thanks for the input and I think you understand the problem very well and why routing info wasn't part of the original format. you are also right in that looking at it in terms of actions is a better approach. Your idea with the relay maintaining a short term cache is a good one and would solve most of the issues everyone is worrying about. It can actually be useful also for sites that want to do day feeds to people. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.07 15:49:00 -
[68] - Quote
Need to use POST for the reasons found in this thread: http://stackoverflow.com/questions/630453/put-vs-post-in-rest To sum up since it's not create a new resource but modifying part of an existing one PUT does not make sense.
What we are missing is a common name for the key. How about UUDIF? So as an example: UUDIF={ "resultType" : "orders", "version" : "0.1alpha", "uploadKeys" : [{ "name" : "emk", "key" : "abc" }, { "name" : "ec" , "key" : "def" }], ...
That should make all the parsers happy.
Edit: You all were posting while I was write stuff I agree if we can work without having to do the whole key-value thing it would be nice but most libraries do seem to prefer having them and tend to make you work with the raw data directly without it which is not so nice Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.07 17:28:00 -
[69] - Quote
I just know that some platforms have limits on length of names etc and thought it would be unlikely to be confused with something else. If we start adding other parameters like data=,message= we might as well not use the format to start with. We only need a single key-value to make it easier to work with for most parsers and it would be helpful to all use the same thing but beyond that it really doesn't matter what it is. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.08 05:19:00 -
[70] - Quote
Ok I guess it wasn't clear from my link why PUT would be wrong so I'll try to explain it. PUT is for creating or updating an existing end point. So if you PUT to SITEROOT/api/item123.xml and can read it back from SITEROOT/api/item123.xml then you can use PUT but if you can't you need to use POST as it's not expected that you can get the data back from the same location with it. If you want to understand this more fully you'll need to read the RFCs related to both POST and PUT.
I updated the Wiki page while I was at it. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
|
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.09 17:31:00 -
[71] - Quote
Been looking into what I need to do to make Yapeal into an uploader for the MarketOrder API and had some questions for the people involved in receiving the data. Because of how the API works it would be nice to send all the market items together in a single POST request just with multiple key-value pairs. I know there was some talk about there being no reason it can't be done from a format point of view but I was wondering if any of you have thought about allowing it and processing them that way. I'll try to make it a little clear how the POST would look: UUDIF={JSON ...}&UUDIF={JSON ...}
Each one would be for a single TypeID so like the first one is TypeID=123 and the second one would be for TypeID=456 etc. so each is a complete dataset they would just be sent together in the same POST.
Is this something you think you'd support or will I need to look at sending a POST for each one? I can see this saving a lot of network bandwidth for everyone if it was supported. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.09 23:20:00 -
[72] - Quote
Guess I need to remember to hit refresh once in a while on that page so I see the changes Ok looks good and that does answer my question Thanks to whomever took the time for the updating on the wiki.
Quote:Most web servers support optional gzipping. Does settiing the content encoding keep it optional? Is there a good argument for making it mandatory? Setting the header saying you accept or would like to send your data compressed it doesn't make it required it just let's the server know that you are connecting to you can and if the server is setup to do so it usually will but it does take both ends agreeing to do so at the start of the connect to happen. If you are using something like cURL there are settings you can use to sent the correct headers and Apache etc have settings to turn it off and on as well.
Edit: The Implementations page was also a good idea Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.10 16:15:00 -
[73] - Quote
If you don't care about checking the header or checksum on PHP you can use method given in http://us3.php.net/manual/en/function.gzdecode.php#106397 instead of the more complex function Desmont McCallock linked. The other function seems to be more for dealing with a group of compressed files or a more complex header then you seem to get from gzencode() or would expect to see in HTTP. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.12 04:19:00 -
[74] - Quote
I deleted the stuff on GET again I don't know who put that back in I dropped it before this last big edit someone did but they added it back with it it seems Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.12 17:01:00 -
[75] - Quote
np I just figured it was someone that had missed when we decided it should be dropped here a page or two back Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.16 22:01:00 -
[76] - Quote
I have no problem with doing so just thought someone had been using it so I didn't take it out when I was doing the edit. You are correct it shouldn't be used for this. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.21 15:32:00 -
[77] - Quote
I think so Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
41
|
Posted - 2012.05.29 15:44:00 -
[78] - Quote
Really this is outside of what the format is about and I guess I don't understand why people keep thinking they need to reinvent the wheel on these things. Everyone keeps getting hung up on how the message is transported between end points but that has nothing to do with the actual format. How it sent between the end points doesn't matter truly. It could be sent through share memory between two process on the same computer, over a serial cable, or bluetooth, etc that is outside of the message. If data is being sent over a TCP/IP connection which has flow control etc that should be used. If it is being sent on port 80 it should follow HTTP rules as well which means when a receiver is down it should return one of the HTTP error codes.
You don't need to 'discover' anything you simple need to sent it and if where you are sending it to accepts it as far as the transmitter is concerned it's done. If the receiver is accepting the data but not doing what it is suppose to that is on them. It's the responsibility of the site you are sending to to make sure when they aren't able to receive the message to response accordingly with what ever flow control the connection has. At the same time you shouldn't be trying to send the message to a site that hasn't clearly said they accept data in the format either as it's just a waste of time for both of you.
Quote:I want to raise a question. What is the reason to use two different outlets for the same data? The idea was tied in with there being mostly sources and sunks and having a channel for sunks to cross talk or the use of relays. Mixed in with that was the idea to allow both push and pull for the syndicated collected data. So the idea was sources would send to api/upload and then the sunk could either forward the data to subscribers automatically through api/syndicate (push) or request it (pull) Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Dragonaire
Corax. The Big Dirty
44
|
Posted - 2012.06.02 14:27:00 -
[79] - Quote
Thank you for starting it all there was diffidently a need for something like this.
How about a link to the main wiki page so it's easier for people to find
http://dev.eve-central.com/unifieduploader/start Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
|
|
|