Pages: 1 2 3 4 5 [6] 7 8 9 10 11 .. 11 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.14 06:52:00 -
[151] - Quote
Ok I've updated the wiki page descriptions to match the new format.
I do have one question though and that is on the 'generatedAt' date/time what it's used for as it's not totally clear the reason for it being there to me at least. Maybe whomever added it can explain it a little bit? Or we can at least define how it's going to be used and maybe change the name so it's use is more apparent without needed to find it in the wiki etc. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Tonto Auri
Vhero' Multipurpose Corp
18
|
Posted - 2011.12.14 22:48:00 -
[152] - Quote
The "generatedAt" could be moved to HTTP Date or Last-Modified header. (If you/your server care to fill it.) For it's purpose, it could be used in the scenario II, where you compile data off hand and present it to clients as cached payload. But as I said, you can do it the straightforward way by utilising HTTP headers.
Sorry for my slow replies in last few days... Nature of my work is that I'm either free for days, even weeks, and then roof crash/hamsters die and I get full hands of work for several hours at least. This time is was full days >.< I'll be back, I still have to write protocol explanation, which forum ate last time and I haven't had a time to repost. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.15 17:29:00 -
[153] - Quote
Quote:I'll be back, I still have to write protocol explanation, which forum ate last time and I haven't had a time to repost. Sorry to hear it got eat I've had that happen a couple times too I'll look forward to reading it when you get back to it.
Quote: The "generatedAt" could be moved to HTTP Date or Last-Modified header. (If you/your server care to fill it.) For it's purpose, it could be used in the scenario II, where you compile data off hand and present it to clients as cached payload. But as I said, you can do it the straightforward way by utilising HTTP headers.
That was kind of my thought. That's in part why I decided to ask the question because I don't really see where having another date/time in the message is adding any value but is just using up extra bandwidth. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
39
|
Posted - 2011.12.15 20:09:00 -
[154] - Quote
Having a HTTP Date or Last-Modified header would only work if you were to send each message individually. I rather think that if you are going to bunch up order or other data, you will want to know when each was generated. Developer/Creator of EVE Marketeer
|
Kaladr
Dreddit Test Alliance Please Ignore
13
|
Posted - 2011.12.15 22:17:00 -
[155] - Quote
Since we're mixing data into one envelope, having the generatedAt per message is very helpful (they could be minute(s) apart) Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Tonto Auri
Vhero' Multipurpose Corp
18
|
Posted - 2011.12.16 08:14:00 -
[156] - Quote
Callean Drevus wrote:Having a HTTP Date or Last-Modified header would only work if you were to send each message individually. I rather think that if you are going to bunch up order or other data, you will want to know when each was generated.
Kaladr wrote:Since we're mixing data into one envelope, having the generatedAt per message is very helpful (they could be minute(s) apart)
1. You can't reliable obtain this information anywhere, unless you export these orders yourself, which, for web-based application, is highly unlikely. 2. Orders list have all the necessary information already (posted date, amount issued, amount left). |
Shirah Yuri
Allied Assault Universal Constant Alliance
12
|
Posted - 2011.12.16 08:30:00 -
[157] - Quote
Tonto Auri wrote:2. Orders list have all the necessary information already (posted date, amount issued, amount left).
For orders, not only it is interesting when the order was posted, but also when it was observed. This information can be used as an offset for all order dates when trying to find out current market activity. Personally, especially when multiple order groups can be transferred, I would prefer a timestamp like this "generatedAt" associated with each one, fully concurring with Kaladr. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.16 09:03:00 -
[158] - Quote
Ok I now better understand where for the aggregators having data on when the rows were collected is important so I have no problem with 'generatedAt' I just wanted to see why it was needed.
Quote:I just noticed the duplication of the columns item though, wouldn't it be better to take that one out of the resultset? Duplicating that will probably happen way more often than duplication of the header content (if you wanted to send multiple result types in one message), though it will also allow for a lot of flexibily to what kind of messages you can put in. You are right that having 'columns' in each of the 'results' is very flexible but also a lot of overhead. The question is does anyone really need the much flexibility and if not how to fix it.
We could bump columns up directly under 'results' but then we'll need to add another container like 'rowset' to hold the array of rows. That would work but still leaves the option open to add or move the columns around but only at the message level which is probably all that would be needed.
Example: "results" : { "columns" : ["price","volRemaining","range","orderID","volEntered","minVolume","bid","issueDate","duration","stationID","solarSystemID"], "rowset" : [ { "generatedAt" : "2011-10-22T15:43:00+00:00", "regionID" : 10000065, "typeID" : 11134, "rows" : [ [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038], [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null], [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039] ] }, { "regionID" : null, "typeID" : 11135, "generatedAt" : "2011-10-22T15:42:00+00:00", "rows" : [ [8999,1,32767,2363806077,1,1,false,"2011-12-03T08:10:59+00:00",90,60008692,30005038], [11499.99,10,32767,2363915657,10,1,false,"2011-12-03T10:53:26+00:00",90,60006970,null], [11500,48,32767,2363413004,50,1,false,"2011-12-02T22:44:01+00:00",90,60006967,30005039] ] } ] }
There are some problems with my example but should we try something like that?
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.16 16:19:00 -
[159] - Quote
After a little sleep and thought I figured out what didn't seem right with the above example. 'columns' needs to be moved up into main part of the message not just up under 'results'. Also thought 'rowsets' would be a better name now. I made the changes on the wiki instead of trying to fight with the forums to show the changes Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
39
|
Posted - 2011.12.16 19:49:00 -
[160] - Quote
I rather think your first message was more in the right direction, as it would still allow for content of different types to be sent in the same message (with a few modifications). But having only one columns header per content type might save some/much space already. Developer/Creator of EVE Marketeer
|
|
Dragonaire
Corax. PURgE Alliance
23
|
Posted - 2011.12.16 23:23:00 -
[161] - Quote
This structure does save a lot of overhead but if anyone can really show where the versatility of the other way would be needed we can look at it. The other thing is the 'resultType' would have to be moved into the rowsets as well to mix order and history together which no one seems to have asked for before.
I just didn't see anyone really needing to mix order and history in the same message anyway. To go along with that there's nothing stopping you from including more than one message in a response as long as both parties want to do that and actually that will probably happen in the case of the pushes since it makes sense to just make a single connection to the receiver and send all the data to them when you can. I even see a case for the uploaders to make a connection and keep it open and send the scan results without having to reconnect all the time.
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Kaladr
Dreddit Test Alliance Please Ignore
13
|
Posted - 2011.12.20 19:20:00 -
[162] - Quote
Dragonaire wrote:This structure does save a lot of overhead but if anyone can really show where the versatility of the other way would be needed we can look at it. The other thing is the 'resultType' would have to be moved into the rowsets as well to mix order and history together which no one seems to have asked for before.
I just didn't see anyone really needing to mix order and history in the same message anyway. To go along with that there's nothing stopping you from including more than one message in a response as long as both parties want to do that and actually that will probably happen in the case of the pushes since it makes sense to just make a single connection to the receiver and send all the data to them when you can. I even see a case for the uploaders to make a connection and keep it open and send the scan results without having to reconnect all the time.
I'd say keeping it at one type per message is preferable right now. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.08 00:05:00 -
[163] - Quote
Thought I'd bump this back up to the top again before it got lost. Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Vaerah Vahrokha
Vahrokh Consulting
109
|
Posted - 2012.01.08 01:26:00 -
[164] - Quote
Callean Drevus wrote:I still cannot understand how his database (EVE Central) is able to handle so much more brute forcing abuse than mine, while he undoubtedly gets a much higher data volume... someone tell me again that this is not simply due to him using PostgreSQL
PostgreSQL is on another league compared to some versions of MySQL.
Also if you search in the PostgreSQL past and roots, you'll find how it's related with a number of other DB engines. Some of these derivatives I used to develop mission critical military software for, that would easily support > 1k concurrent users (Solaris on Ultra 4000) with a downtime of about 1 day per year. That day was used to upgrade the back end, not because the engine needed any downtime ever. |
Steve Ronuken
Fuzzwork Enterprises
152
|
Posted - 2012.01.08 02:57:00 -
[165] - Quote
Mysql is pretty poor on scaling.
Postgres is better, though you'll want some form of connection pooling.
If you are using mysql I'd suggest making sure you have result caching turned on, and batch your updates (so you don't constantly invalidate the cache) FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator and other 'useful' utilities. |
Scrapyard Bob
EVE University Ivy League
586
|
Posted - 2012.01.08 04:48:00 -
[166] - Quote
pgsql also benefits hugely from putting the transaction log on a separate spindle set from the main database, especially if you are doing heavy inserts (of millions of rows).
But mostly it just works and works well. The defaults are much better tuned in versions past 8.2-ish. |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.08 07:54:00 -
[167] - Quote
Using tokuDb instead of normal MyISAM or InnoDB tables is a great idea if you are dealing with really large tables. http://www.tokutek.com/ I've done some testing even on fairly small data sets and it's much better there too.
I also prefer to use MariaDB over normal MySQL as it seems to work better and has less bugs in the main code. http://mariadb.org/ Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Kaladr
Dreddit Test Alliance Please Ignore
18
|
Posted - 2012.01.09 17:47:00 -
[168] - Quote
Vaerah Vahrokha wrote:Callean Drevus wrote:I still cannot understand how his database (EVE Central) is able to handle so much more brute forcing abuse than mine, while he undoubtedly gets a much higher data volume... someone tell me again that this is not simply due to him using PostgreSQL PostgreSQL is on another league compared to some versions of MySQL. Also if you search in the PostgreSQL past and roots, you'll find how it's related with a number of other DB engines. Some of these derivatives I used to develop mission critical military software for, that would easily support > 1k concurrent users (Solaris on Ultra 4000) with a downtime of about 1 day per year. That day was used to upgrade the back end, not because the engine needed any downtime ever.
Some of it is PostgreSQL, some of it is heavy-handed caching, some of it is luck (maybe? )
Honestly, the biggest limit is disk bandwidth, which is a limit for writes (and impacts reads). You can cheat and run PostgreSQL in "NoSQL mode" (turn off synchronous commits), which erodes at the durability/server shutdown survivability.
The EC active dataset can be cached in RAM fairly easily (which is why I run with no less than 16GB, and the new system runs at 48GB), which helps read performance. Believe it or not, I am actually running into CPU contention now... Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
27
|
Posted - 2012.01.09 18:21:00 -
[169] - Quote
Everyone I know of that uses Yapeal in large deployments runs into the same problems with network I/O (rarely), DB bandwidth which when you look into it in depth turns out to be Disk I/O problems 99.9% of the time. Most shared hosting has very poor disk I/O so lots of memory to handle surges is a must but at some point it has to go to disk and then your 3GHz 8 way server just became a 50Mbps server in most cases Finds camping stations from the inside much easier. Designer of [url]http://code.google.com/p/yapeal/[/url] for Eve API. Check out the [url]https://forums.eveonline.com/default.aspx?g=posts&t=7540[/url] |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
64
|
Posted - 2012.01.10 18:59:00 -
[170] - Quote
I can assure you that EMK is not running on a shared host (anymore) . The disk IO was a problem before, but since moving to Storm on Demand it hasn't been an issue anymore. Currently I'm just running into the limits of either my MySQL configuration knowledge, or the inherent problems in MySQL (or TokuDB, which I'm using now). The problem is the cache is invalidated most of the time due to updates. I'd rather update in memory, but that leaves the previously mentioned problem of data durability.
The feature I'm mostly talking about is the Trade Finder by the way. And I just found out last week that part of my speed issues were due to my insistence to do the whole calculation in the database, while EC actually does a lot in python instead.
Aside from that, I think it would be a good time to start implementing the new upload format now. Or at least, in the foreseeable future. I think we're mostly done with modifications for now. Developer/Creator of EVE Marketeer
|
|
Kaladr
Dreddit Test Alliance Please Ignore
18
|
Posted - 2012.01.10 21:21:00 -
[171] - Quote
EC does the "buy/sell" trade calculation in a (very unmaintainable) SQL query. The path finding is farmed out to C++, and the whole deal is sorted in Python.
The "sell/sell" trade finder is managed in Python.
The new buy/sell trade finder is now in Scala similar to how the sell/sell one worked before. Once Hotspot has had a pass at it its nice and speedy. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Garo Hertee
Eclipse Industrials Quantum Forge
14
|
Posted - 2012.01.11 00:05:00 -
[172] - Quote
An excellent week for the Eve market sites...
Eve Central: 100% uptime Eve Market Data: 100% uptime Eve Marketeer: 100% uptime
Congratulations guys. |
Dragonaire
Corax. The Big Dirty
35
|
Posted - 2012.03.12 01:25:00 -
[173] - Quote
Just thought it would be a good idea to bump this back to the top again Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Desmont McCallock
143
|
Posted - 2012.03.20 07:35:00 -
[174] - Quote
Just a quick question. Why is 'issuedDate' being parsed into DateTime format and not use the original *NIX timestamp (which is actually how the data have it)? Is this some kind of DB restriction? |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
114
|
Posted - 2012.03.20 20:39:00 -
[175] - Quote
issuedDate in the EVE cache is actually a strange windows time format, that absolutely has to be parsed, as far as I have been able to understand.
At least, I assume that is the one you are talking about. Developer/Creator of EVE Marketeer
|
Desmont McCallock
146
|
Posted - 2012.03.20 21:22:00 -
[176] - Quote
Callean Drevus wrote:issuedDate in the EVE cache is actually a strange windows time format, that absolutely has to be parsed, as far as I have been able to understand.
At least, I assume that is the one you are talking about.
It's actually *NIX timestamp expressed in ticks. |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
114
|
Posted - 2012.03.21 09:30:00 -
[177] - Quote
Desmont McCallock wrote:I'll get into it further tomorrow as time is late and my head is spinning atm. Edit: After some researching the 'issuedDate' is expressed in Windows NT time ticks, which has an initial datetime of '1601/1/1 00:00:00' much like *NIX timestamp has '1970/1/1 00:00:00'. For more info visit: http://support.citrix.com/article/CTX109645So is it possible to upload the data with the Win NT timestamp and the parsing to be done on the server?
Sure, as far as I know, but why give the server even more to do when it can just as easily be done on the client, and leave you with a more readable format when sending? I know I'd much rather see a real date than a string of numbers.
EDIT: It just occurred to me that parsing a datetime string is probably more of a hurdle for the server, but I still vote that over an esoteric numeric format, that everyone will think is a timestamp, while it really isn't. Developer/Creator of EVE Marketeer
|
Desmont McCallock
148
|
Posted - 2012.03.21 11:22:00 -
[178] - Quote
TBH I don't run a server and I'm not so familiar with maintaining a DB. But parsing the WinNT timestamp isn't much of a hurdle. After all it's a one liner code.
NVTL, what does Kaladr and Dragonaire have to say about it?
@Callean And while you are reading this, are you going to support the unified format? |
Dragonaire
Corax. The Big Dirty
35
|
Posted - 2012.03.21 16:49:00 -
[179] - Quote
Desmont McCallock wrote:Just a quick question. Why is 'issuedDate' being parsed into DateTime format and not use the original *NIX timestamp (which is actually how the data have it)? Is this some kind of DB restriction? This was discussed some where earlier in the thread but I'm not finding it right now. The reason we choose the current format is it is easy to understand for people and computers. It's not OS dependent and most databases can work with it directly or it can be translate easily for those that don't. Most programming languages also can handle it with little trouble when need and more important since we're developing a 'standard' it makes much more sense to leverage other existing ones that have been well thought out, peer reviewed, and is well documented.
One thing to keep in mind is this is a data exchange format and not a data storage format and as originally stated we wanted it to be understandable even by a programmer that had never seen it before without having to read a bunch of docs somewhere. Much like JSON itself being able to describe it with a single page and just a couple of examples with descriptions was at least part of my original concept and I believe everyone else's. We also wanted something that was compact but not to the point that it becomes hard to understand or use. There was some early talk about using XML but it's just to bloated for something like this so that's where JSON won out.
Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal PHP API library thread for more information. |
Desmont McCallock
148
|
Posted - 2012.03.21 17:05:00 -
[180] - Quote
Sure, the logic behind the design of the unified format is unquestionable and my question was more than trivial.
The real question here is will the market sites (at least those that are receiving atm data) support that format?
Coding the EVEMon market uploader (with support for custom entered endpoints) will depend upon that implementation. |
|
|
|
|
Pages: 1 2 3 4 5 [6] 7 8 9 10 11 .. 11 :: one page |
First page | Previous page | Next page | Last page |