Pages: 1 2 3 [4] 5 6 7 8 9 10 11 .. 11 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 17:01:00 -
[91] - Quote
Ok after looking at both the Eve cache export data and Eve API MarketOrder columns I revised the 'orders' JSON format to use the same names etc in the wiki. This should allow developers from both areas to use it with minimal amount of data conversion. I used the same order for the columns as given by cache export data.
In the 'history' I decided to use the same order of columns as shown in game. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 00:50:00 -
[92] - Quote
I'm really sorry about the triple post but I thought this was important enough and unrelated enough to my others to use one.
I'll direct you to a new forum thread on a new project those developers reading this thread might find interesting. https://forums.eveonline.com/default.aspx?g=posts&t=4169 Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Desmont McCallock
52
|
Posted - 2011.12.04 08:07:00 -
[93] - Quote
Linky leads to.... WH 404? |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 08:36:00 -
[94] - Quote
Sorry I miss edited it it should work now. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
38
|
Posted - 2011.12.04 09:56:00 -
[95] - Quote
Ok, so I just found out I have lost the source code to the EMK uploader in a crash last week. Unless I can decompile the exe, I'll be stuck having to build it all again (including threaded support of course)... Developer/Creator of EVE Marketeer [url]https://chrome.google.com/webstore/detail/dfnemlggfgadocgjllogmcbihmedhjao[/url] |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 10:07:00 -
[96] - Quote
Was it in hg? If so you should still have it in one of the other copies out there. I might even have an older one floating around somewhere Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
38
|
Posted - 2011.12.04 12:26:00 -
[97] - Quote
I wish :P that's the problem. I totally forgot to ever put that thing in version control. Developer/Creator of EVE Marketeer [url]https://chrome.google.com/webstore/detail/dfnemlggfgadocgjllogmcbihmedhjao[/url] |
Tonto Auri
Vhero' Multipurpose Corp
15
|
Posted - 2011.12.04 16:17:00 -
[98] - Quote
Callean Drevus wrote:I wish :P that's the problem. I totally forgot to ever put that thing in version control. O.o This is usually the second step I take for my own projects. I have a server next to me, with Subversion running over mirror raid. |
Desmont McCallock
52
|
Posted - 2011.12.04 18:08:00 -
[99] - Quote
Callean Drevus wrote:Ok, so I just found out I have lost the source code to the EMK uploader in a crash last week. Unless I can decompile the exe, I'll be stuck having to build it all again (including threaded support of course)...
I thought I had a copy but I was wrong. As you said you never put it in VCS so I couldn't have one. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 18:54:00 -
[100] - Quote
So been think on the JSON order feed stuff a little more and wondering if it really needs to include solarSystemID since that can be inferred from the stationID. The same could be said for the regionID but since it's only used once in the metaData it's less of an issue IMHO.
Anyone else have a feeling about this one way or the other?
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
38
|
Posted - 2011.12.04 19:36:00 -
[101] - Quote
I think solarSystemID is not necessary, on the other hand, it's very convenient, as it saves a query to the database every upload :)
Quote:O.o This is usually the second step I take for my own projects. I have a server next to me, with Subversion running over mirror raid.
Indeed, I normally also have an offsite backup, but this time it seems I got so caught up in developing the thing that I forgot it (I normally do my copy to live by way of version control, so it's impossible to forget, but desktop applications are packaged). Developer/Creator of EVE Marketeer
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 19:53:00 -
[102] - Quote
Guess it comes down to how normalized your DB is but since orders always happen at stations and they don't move from solar system to solar system there's really no reason to be saving it to the DB and only need to use join on selects which should be being cached in most cases anyway. If you're trying to keep some common per region or solar system stats table up to date it should be ran at the end of the feed insert/update so it's only running once per feed not on each row which would be very slow. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
15
|
Posted - 2011.12.04 21:17:00 -
[103] - Quote
Dragonaire wrote:So been think on the JSON order feed stuff a little more and wondering if it really needs to include solarSystemID since that can be inferred from the stationID. The same could be said for the regionID but since it's only used once in the metaData it's less of an issue IMHO.
Anyone else have a feeling about this one way or the other?
The main question is: Do you need it to know at the insertion stage? |
Kaladr
Dreddit Test Alliance Please Ignore
11
|
Posted - 2011.12.04 21:22:00 -
[104] - Quote
I made one minor change: columns is now a list instead of just a string. No reason to split on a tokenized string if you don't have to.
As for solar system, EVE-Central runs with somewhat denormalized tables, and makes use of solarSystemID. But its not a big loss if it doesn't exist. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 22:01:00 -
[105] - Quote
Main reason I changed columns to be a list was to have it match how it was done in the APIs but either way works fine. Having it as an array would be better in that it'll directly match up with rowset rows.
So after making myself look a bit foolish on another thread the next question I have is should 'currentTime' be formated as in the API with'-' and ':'s or like is done in log file names with '.' and nothing? The API has advantage of never having anyone having the same confusion I did but the other format would let it be extracted directly by cache readers from the file name. It's understandable why CCP didn't use them in file names because they aren't allowed on many OSes or have special means but it purely up to us on the feed. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
15
|
Posted - 2011.12.04 22:32:00 -
[106] - Quote
I'd use one of the standard time formats. Perhaps, ISO 8601 (XML standard date specification, like 2004-02-12T15:19:21+00:00). |
Kaladr
Dreddit Test Alliance Please Ignore
11
|
Posted - 2011.12.04 23:04:00 -
[107] - Quote
Tonto Auri wrote:I'd use one of the standard time formats. Perhaps, ISO 8601 (XML standard date specification, like 2004-02-12T15:19:21+00:00).
+1 for standard formats. While I like the idea of converging on an existing format, the EVE one isn't exactly the best one :) Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.04 23:06:00 -
[108] - Quote
I do wish CCP had used that for the APIs it really would have made everything easier. It might make sense to do that here and just have everyone manage the conversion in the clients.
So there's another related question here about the rowset 'issueDate'. Should it also be converted so all the date-times are in the same format? I think it would be a good idea and barring that since we do have 'issueDate' in the same format as the APIs then 'currentTime' should be the same as it.
I'm up for everything being in ISO 8601 as it would let clients use their local time in 'currentTime' and the times from the API and cache are already in UTC so they can be converted with simple string appending. I'll update the wiki as I think it's a good way to go with this one. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
16
|
Posted - 2011.12.05 00:24:00 -
[109] - Quote
To answer your question, I can give you two immediate thoughts: 1. Using raw data as coming from cache reader, is easier on processing. But. 2. You are not guaranteed, that the data from cache reading would stay in the same format for foreseeable future.
My own opinion: While using human-readable format for rowset data is tempting, usage of more compact representation is advisable for technical reasons. I'd stick with UNIXTIME here. |
Tonto Auri
Vhero' Multipurpose Corp
16
|
Posted - 2011.12.05 00:34:00 -
[110] - Quote
Dragonaire wrote:I also dropped solarSystemID from 'columns' and 'rowset' but that still leaves a issue for me in Yapeal with 'regionID' So I have another Idea what to do for it and 'solarSystemID'. JSON has a null value so what if we allow both 'regionID' and 'solarSystemID' be null or set? That way clients that have access to them can include them but ones that don't can just use null instead. Does this sound like a solution that workable for everybody?
I have no JSON experience myself (a regrettable hole in my education, which I have to plug at some point), but I think your reasoning is sound.
On another note, whoever edited the wiki, please check the first example :) And disable spell checking addons while doing so. (volEntered - volunteered) |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 00:50:00 -
[111] - Quote
Quote:On another note, whoever edited the wiki, please check the first example :) And disable spell checking addons while doing so. (volEntered - volunteered) I don't think it was my edit that did that but I fixed it anyway.
JSON is really easy to understand really the spec for it is like one page. [link url=http://www.json.org/]http://www.json.org/[/link] Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
16
|
Posted - 2011.12.05 02:53:00 -
[112] - Quote
While i'm still around, two more issues to throw in a pot: 1. Data encoding. While there's little to no string data communicated in these requests, I think it's important to make a note that any string data SHOULD be encoded in UTF-8. 2. It needs a recommended algorithm to generate 'unique' field value. Something as "simple" as md5 of a certain string value (f.e., program signature concatenated with datetime of a generated request). But not leave it as "oh, just put something there". |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 04:58:00 -
[113] - Quote
Quote:1. Data encoding. While there's little to no string data communicated in these requests, I think it's important to make a note that any string data SHOULD be encoded in UTF-8. From JSON docs: A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.
Quote:2. It needs a recommended algorithm to generate 'unique' field value. Something as "simple" as md5 of a certain string value (f.e., program signature concatenated with datetime of a generated request). But not leave it as "oh, just put something there". I not sure if I'm understand what you're talking about so please correct me if I'm wrong.
If you are talking about a way to prevent network loops it'll be up to each site to decide how they detect it but some pointers on what fields might be used could be added to the wiki. Since this will be used from upload clients to sites or site to site pulls there shouldn't be any need to include something beyond the existing metadata.
In the case of cache readers they already usually delete the cache files after sending so they don't try to resend the same data. For something based on the API you could use the cachedUntil time.
In the case of site to site transfers it'll be done as pulls from RESTful service where the 'client' well be asking for just the updates they want and will need to decide based on the data if they are going to update their internal data. So for example a request for Amarr Shuttle orders in Kor-Azor would be something like www.example.com/EveApiFeed/Kor-Azor/Amarr+Shuttle/. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 09:46:00 -
[114] - Quote
Ok I made a change to "updateKeys" which should allow people to use the "key" for several different things. Also tried to make it clearer for people what can and can't be used from the JSON message for the "key" here as well. It may still need some refining but at least now everyone that wants to add something 'unique' to the message to ID it can without interfering with anyone else's idea how to do the same thing Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
16
|
Posted - 2011.12.05 13:41:00 -
[115] - Quote
There was an issue with parsing this post's BBCode |
Desmont McCallock
52
|
Posted - 2011.12.05 13:49:00 -
[116] - Quote
I don't know about you but in .NET if you use HttpUtility.UrlEncode it adds "+" as space. On the other hand, if you create a new Uri from the string it uses "%20" as space. |
Tonto Auri
Vhero' Multipurpose Corp
16
|
Posted - 2011.12.05 13:54:00 -
[117] - Quote
Ok, i'm exhausted trying to work around forum stupidity. @Desmont: unless you encode URL parameters, using "+" as space replacement is just wrong. |
Desmont McCallock
52
|
Posted - 2011.12.05 13:56:00 -
[118] - Quote
Tonto Auri wrote:Ok, i'm exhausted trying to work around forum stupidity. @Desmont: unless you encode URL parameters, using "+" as space replacement is just wrong. I agree. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.05 16:39:00 -
[119] - Quote
I take it you guys didn't like my example above I'm sure you are right and it should be done as "%20" so I edited it .
So besides my bad URL what does everyone think of the change I made? I didn't realize there was a problem with how the "uploadKeys" was done before until I was trying to document it and didn't have anything for the Name column in the table. I'd also just finish deleting that whole "unique" thing again and it hit me it might be possible to do something that could be used for either or both without it interfering with anyone else ideas of how these things could be solved. So now we have a "key" where anyone that cares to can either do something neat and useful or just shoot themselves in the foot instead and have minimal impact on anyone else.
A 'BAD' example: Use "key" to hold to complete encrypted message minus "uploadKeys" and "currentTime"
'GOOD' example: Use "key" to hold a simple registered clientID so they can get credit for uploading market order data.
'BETTER' example: Use "key" to hold clientID like in 'GOOD' but also a digest of it and the complete message from the 'BAD' example. "key" : "{clientID}|{digest}"
There are many other possibilities but that should give everyone some ideas about how it could be used or abused. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
38
|
Posted - 2011.12.06 23:05:00 -
[120] - Quote
Enlighten me again why someone decided to drop the uniqueId? I wasn't the one whom replaced it, but I still advocate it's use.
Instead of implementing a silly hashing function on the server, which admittedly, you could implement any way you want, but which would be a failure by default since you would be reimplementing the very unique uniqueness of a UUID.
I personally very much like the idea of only having to compare a list of recently uploaded id's against the id of a new upload to see if I already got it.
That is, any calculation I can get done clientside is welcome, since that saves resources on the server.
And I like the 'good' example, I'll personally eliminate any upload where the key is longer than 50 characters :P Developer/Creator of EVE Marketeer
|
|
|
|
|
Pages: 1 2 3 [4] 5 6 7 8 9 10 11 .. 11 :: one page |
First page | Previous page | Next page | Last page |