Pages: 1 2 [3] 4 5 6 7 8 9 10 11 .. 11 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Nam Noissim
Red Lobsters Unilateral
1
|
Posted - 2011.11.11 22:03:00 -
[61] - Quote
In terms of the config file, if you are writing your uploader in C/C++, i'd recommend iniparser. I use it in my own apps and it is a wonderful pre-made solution. Just download the files, grab out the 4 code files (2.c, 2.h) and include them. It works great.
http://ndevilla.free.fr/iniparser/
If you need an XML parser, I've been using a customized version of http://www.applied-mathematics.net/tools/xmlParser.html this guy's parser for several years now. It is fast, correct, etc. I like it. It isn't a library, just like iniparser. You just take the .h and .c files and compile with them.
What did I tweak in his code? Well, mainly XMLParser decided that if *anything* went wrong, he should exit your application. Yea...no. I went through and stripped all that garbage out and replaced it with more graceful failures my app can catch and deal with. I think I might have changed a couple other things, but that is the big one that I remember annoying me for weeks on end.
Other than that...*shrug* sit down and code. :) |
Garo Hertee
Eclipse Industrials Quantum Forge
5
|
Posted - 2011.11.11 22:18:00 -
[62] - Quote
Either one, as long as the config files are easy for anyone to create with very simple documentation. I'm a web programmer (my forte is PHP) and have never really touched C, but it's pretty easy to document how to write a config file in XML. It would be a process which presumably anyone creating a new market site would be familiar with. |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
33
|
Posted - 2011.11.12 19:13:00 -
[63] - Quote
Garo Hertee wrote:Is Eve Marketeer processing data right now? I'm uploading but...
1. My upload count isn't rising 2. The stats page isn't showing uploads for any region in the past 24 hours 3. The upload backlog is showing almost 68000.
Does this mean it's accepting uploads into a queue but not processing that queue?
It was processing the queue, but not fast enough. That is, I had had the EVE Central import turned off for a while, and when I enabled it again it immediately resumed processing where it left off, filling the queue with about 2 weeks of stale data, which I only just removed in a controlled way.
Quote:To those running the sites, please don't take these figures as a criticism, they're intended purely as figures.
Interesting to see this uptime, even if it doesn't show EVE Marketeer in a very good light it is probably correct. Even if the website is cool, it suffers from a lot of problems and it is mostly in the experimental stage right now. [img]http://www.evemarketeer.com/player/sig/511049420[/img] Want images? |
Garo Hertee
Eclipse Industrials Quantum Forge
6
|
Posted - 2011.11.12 19:20:00 -
[64] - Quote
That's good. The numbers were intended as a baseline which you can start to work from. If it's useful to you, I can post them occasionally to give you an idea of progress. |
Shingihada Rubik
University of Caille Gallente Federation
0
|
Posted - 2011.11.13 16:04:00 -
[65] - Quote
Quick question regarding the EVE Marketeer Uploader, does it send all the data directly to each of the sites or does it queue up locally on your server before being sent to the other sites? Not that I have any clue whatsoever about programming, I'm just curious. |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
33
|
Posted - 2011.11.14 22:52:00 -
[66] - Quote
It sends to every site directly, that eliminates the problem of the single point of failure being my website, which Garo so cheerfully proved to be the most unreliable of all the market websites ;) [img]http://www.evemarketeer.com/player/sig/511049420[/img] Want images? |
Garo Hertee
Eclipse Industrials Quantum Forge
6
|
Posted - 2011.11.16 19:45:00 -
[67] - Quote
LOL, not cheerfully. Anyway, the 100% for Eve Central may not be entirely accurate. When it's not working, it returns a proxy error page and the monitoring system is probably counting this as being a response on port 80, even though the site isn't providing data. |
Garo Hertee
Eclipse Industrials Quantum Forge
7
|
Posted - 2011.11.17 22:47:00 -
[68] - Quote
So here are this week's figures which show that Eve Central doesn't have an automatic 100%...
Eve Central: 99.7% uptime Eve Market Data: 98.2% uptime Eve Marketeer: 85.1% uptime |
Garo Hertee
Eclipse Industrials Quantum Forge
12
|
Posted - 2011.11.23 20:09:00 -
[69] - Quote
Callean, I'm not sure if you've been working on the server to improve stability, but if you have, it's working.
Eve Central: 100% uptime Eve Market Data: 95.9% uptime Eve Marketeer: 92.1% uptime |
Kaladr
Viziam Amarr Empire
4
|
Posted - 2011.11.28 23:34:00 -
[70] - Quote
As the operator of EVE-Central, I always welcome improvements to the Uploader portion. I am also very willing to change the API as needed to have a more unified data format. I am not planning on deprecating the old interface either.
As I've announced elsewhere, EVE-Central is still interesting to me and am recently upping my contribution back to the site (and EVE in general, though my picture still seems to be stuck in 2008 <--- ).
To those of you who have created uploaders already, thanks! Contribtastic has some annoying bugs, and it would be great to pave over them, but honestly client side UIs have never been my strong suit.
As for the more "exclusive" uploaders, it would be great to have a federated data feed amongst ourselves. The SMTP service from EVE-Central is not the best tool (but it was admittedly easy to hack together), but something similar would always be appreciated. |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.29 05:25:00 -
[71] - Quote
I was going to suggest something like RSS with JSON payload but after looking into RSS a little more it seems it can't handle what is needed so maybe just straight forward RESTful web application type interface might be the way to go. The format I suggested back in post #36 might be usable directly or with a few changes. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Kaladr
Viziam Amarr Empire
5
|
Posted - 2011.11.29 05:35:00 -
[72] - Quote
Dragonaire wrote:I was going to suggest something like RSS with JSON payload but after looking into RSS a little more it seems it can't handle what is needed so maybe just straight forward RESTful web application type interface might be the way to go. The format I suggested back in post #36 might be usable directly or with a few changes.
RESTful applications are the way to go. The end-point doesn't have to be consistent as long as the payload is consistent - I'm perfectly fine with your suggested JSON payload, assuming it would be generated per site, otherwise the developer key portion would need multiple values per site.
Awhile ago EVE-Central played with the notion of sucking in API orders information and transactions. For EVE-API stability reasons this was disabled, but I'm very tempted to turn that back on again (with an improved user login system). Its only a minor snapshot of the world, and is more easily outdated than cache scrapes, but added some useful information. Sadly that information was not previously syndicated (though there is no technical reason it couldn't be). Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.29 22:47:00 -
[73] - Quote
I've also looked at adding the same thing to Yapeal to grab market order information but some other things have had a higher priority lately (SkyRim) for me
The idea with developer_key was to have some kind of site key that others could check mostly just to have an idea where it came from to prevent loops. It could probably be better named. I thought about having some kind of two part encryption system but then we'd have to deal with the whole trust issue and I thought it would be better let each site decide how they wished to handle that part but I think there needs to be some way inside the meta data to track at least where it says it came from. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Kaladr
Viziam Amarr Empire
5
|
Posted - 2011.11.29 23:50:00 -
[74] - Quote
Dragonaire wrote:I've also looked at adding the same thing to Yapeal to grab market order information but some other things have had a higher priority lately (SkyRim) for me The idea with developer_key was to have some kind of site key that others could check mostly just to have an idea where it came from to prevent loops. It could probably be better named. I thought about having some kind of two part encryption system but then we'd have to deal with the whole trust issue and I thought it would be better let each site decide how they wished to handle that part but I think there needs to be some way inside the meta data to track at least where it says it came from.
Ok, not a bad idea on that end - what you're really looking for is a UUID of some kind, to avoid circularly publishing data. I'm all for that, and it should probably be client generated. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Kaladr
Viziam Amarr Empire
5
|
Posted - 2011.11.30 07:03:00 -
[75] - Quote
Ok, I have taken the EMK format, massaged it a bit, and tossed it up on the EVE-Central Development Wiki (hint: anyone can edit this).
http://dev.eve-central.com/unifieduploader/start
Comments?
At present (or for at least a year) not deprecate the old "throw CSV at server" upload system in EVE-Central, since there are now multiple clients which reference it. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
36
|
Posted - 2011.11.30 08:14:00 -
[76] - Quote
Hey, good to see the creator of EVE Central return.
Besides, with that upload format, I really can't say anything for CSV. It looks pretty decent, with useful JSON in the places where it is relevant, but leaving the orders part an array, which will save on bandwidth and be more concise in general.
It also includes ALL data in the POST part of the request, which is pretty nice (no more parameters). UUID's to uniquely identify uploads are a pretty good idea, and not one I ever thought of (though I'd never heard of the concept until a few months back).
Today I'm going to modify the uploader to hopefully use seperate threads and allow arbitrary upload locations, and then we'll really be on our way towards a unified upload infrastructure.
In addition, I"ve been planning for a while to implement upload queueing in EMK, and that will be the way uploads can be posted to other sites/services/anyone as well (using the same upload API the clientside uploaders use). [img]http://www.evemarketeer.com/player/sig/511049420[/img] Want images? |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.30 16:29:00 -
[77] - Quote
@Kaladr Linky no worky for me. It just times out but even www.eve-central.com is being really slow right now too and is probably going to time out. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Kaladr
Viziam Amarr Empire
6
|
Posted - 2011.11.30 18:16:00 -
[78] - Quote
Dragonaire wrote:@Kaladr Linky no worky for me. It just times out but even www.eve-central.com is being really slow right now too and is probably going to time out.
Well there goes two weeks of uptime due to an odd bug.. Backend servers punted and back to normal.
New server should be in sometime in January to help scale out. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 07:07:00 -
[79] - Quote
Like your ideas but I think it would be a good idea to still do the data part with the extra header stuff from my improved version.
"log":{ "header":["order_id","buysell","solar_system_id","station_id","price", "vol_entered","vol_remaining","min_volume","issue_datetime","duration", "range"], "rows":[ [...] ]
It would make it more self documenting which would allow developers that weren't involved in this thread to still understand what everything was and also makes any future changes easier if needed to add additional columns etc. It would also let developers have more freedom if needed in the order of the columns should it be ever be required in their application. And then there is of course the most important reason it makes it easier for anyone (read me) when I'm trying to fix something that broke at 3am and I haven't had any sleep for 3 days so it's hard for me to remember which came first vol_entered or vol_remaining etc The extra overhead is minimal but adds a lot of value IMHO.
I do like the idea of maybe using UUIDs but how would it be better than just using a MD5 or SHA1 hash of the URL with parameters? Also which parameters were you thinking about using/including in it?
BTW thanks for fixing it so I could see the linky Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Desmont McCallock
51
|
Posted - 2011.12.01 07:42:00 -
[80] - Quote
Kaladr, what Drago says. CCP makes our lives difficult as it is. Let's not make them more. |
|
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 23:27:00 -
[81] - Quote
I updated the wiki so everyone can take a look and see if they have any other changes on it. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Garo Hertee
Eclipse Industrials Quantum Forge
12
|
Posted - 2011.12.02 21:03:00 -
[82] - Quote
A good week for the Eve market sites...
Eve Central: 100% uptime Eve Market Data: 100% uptime Eve Marketeer: 98.2% uptime |
Kaladr
Viziam Amarr Empire
8
|
Posted - 2011.12.02 21:27:00 -
[83] - Quote
Dragonaire wrote:I updated the wiki so everyone can take a look and see if they have any other changes on it.
Looking good so far. I made one additional change: the inclusion of a generated_by map which lists the upload client name and version.
I am currently working on updates to Contribtastic (and creating Contribtastic Mac), and will look at making a version which can generate this upload format. Creator of EVE-Central.com, the longest running EVE Market Aggregator |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 03:12:00 -
[84] - Quote
generated_by is a good addition don't know why I hadn't thought of it. I added a history example for us to work with as well.
Quote:I do like the idea of maybe using UUIDs but how would it be better than just using a MD5 or SHA1 hash of the URL with parameters? Also which parameters were you thinking about using/including in it? Still wondering about this. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
37
|
Posted - 2011.12.03 10:02:00 -
[85] - Quote
Well, the UUID wouldn't be based on any fields, it would just be generated based on the natural randomness of the system (or using the fields, if you feel so inclined), the point is that the value of a UUID isn't based on an input that was given it, so it's almost guaranteed to be unique (much more so than MD5 for example, which could collide quite a bit more, even if you were using the unique URLS to generate it).
That at least, is the great plus of UUID in my opinion. Maybe someone else had some other good reasons to use it.
UPDATE: In addtion, if we are going to use that header row. I'd call it 'fields' or 'data_fields' instead of 'header' as it doesn't really have much to do with headers, and we might as well be semantically correct :) [img]http://www.evemarketeer.com/player/sig/511049420[/img] Want images? |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:11:00 -
[86] - Quote
Quote:UPDATE: In addtion, if we are going to use that header row. I'd call it 'fields' or 'data_fields' instead of 'header' as it doesn't really have much to do with headers, and we might as well be semantically correct :) The reason I originally named it 'header' was I was viewing it like the header line of a CSV file but it could be named differently. Maybe 'field_names' or 'column_names' would be a better name.
The naming that really bothers me more is 'log'. I used that because the original uploader examples I looked at used that but IMHO naming it 'data', 'contents', or 'payload' would make more sense.
Quote:... That at least, is the great plus of UUID in my opinion. Maybe someone else had some other good reasons to use it. Actually UUID version 3 and 5 are MD5 or SHA1 of the URL so there isn't really much difference except for the formating that was why I asking the question since it would be possibly a extra step without really adding any extra value.
I've just been giving this a lot more thought and I think we might be going at this the wrong way. Instead of trying to figure out a common digest or checksum let's simple make sure to include enough metadata so that we know the receiver will have everything they would need to make one to insure loops don't happen and leave it up to the developers what they use. The network bandwidth and memory buffering differences is nil as long as it's less than a single TCP packet worth of data or at most two for locations with a lot of data rows.
Looking at it from the network bandwidth angle it might be a good idea to think a little more about how stuff is named but we don't want to go to far trying to make it smaller and end up make it human unfriendly either. I'll look at changing some of the names on the wiki and see what everyone thinks. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Desmont McCallock
52
|
Posted - 2011.12.03 14:21:00 -
[87] - Quote
Dragonaire wrote:Quote:UPDATE: In addtion, if we are going to use that header row. I'd call it 'fields' or 'data_fields' instead of 'header' as it doesn't really have much to do with headers, and we might as well be semantically correct :) The reason I originally named it 'header' was I was viewing it like the header line of a CSV file but it could be named differently. Maybe 'field_names' or 'column_names' would be a better name. The naming that really bothers me more is 'log'. I used that because the original uploader examples I looked at used that but IMHO naming it 'data', 'contents', or 'payload' would make more sense. I would go for : 1. header -> columns 2. log -> rowset |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:29:00 -
[88] - Quote
Would bring it more in line with API XML which might have some benefits on developer understanding and code reuse maybe. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Desmont McCallock
52
|
Posted - 2011.12.03 14:31:00 -
[89] - Quote
Dragonaire wrote:Would bring it more in line with API XML which might have some benefits on developer understanding and code reuse maybe. Exactly my point. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.03 14:36:00 -
[90] - Quote
Should we go the rest of the way and call it 'result' and use currentTime too? Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
|
|
|
Pages: 1 2 [3] 4 5 6 7 8 9 10 11 .. 11 :: one page |
First page | Previous page | Next page | Last page |