| Pages: 1 2 3 [4] 5 6 7 8 9 :: one page |
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |

Snarf Aldes
University of Caille Gallente Federation
6
|
Posted - 2012.04.23 10:19:00 -
[91] - Quote
Ilyk Halibut wrote:Snarf Aldes wrote:Question.. Why are you reformatting all the incoming data? I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID. Here is an example split_json.txtThe top is what i am sending, the bottom is what is returned through EMDR. I really need it to be in the top format, i cannot use the bottom format  Sorry about this, I hunted it down and corrected this just now. It won't be a problem anymore. Cool, I switched the EMDR stream on again. I am seeing duplicates for every entry now, but that could be because the same data is originating from different sources. Creator of Eve Addicts |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.23 14:10:00 -
[92] - Quote
Snarf Aldes wrote: Cool, I switched the EMDR stream on again. I am seeing duplicates for every entry now, but that could be because the same data is originating from different sources.
Yeah, you should see up to three duplicates, since the Unified Uploader uploads to EVE Central, EVE Marketeer, and EVE Marketdata. We just started getting redirected downloads from EVE Marketeer, so this is more noticeable, since we cover all three.
This bothered me initially, but it does allow consumers to cross-check data to make sure some dufus like Milla isn't uploading bad data. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.24 04:21:00 -
[93] - Quote
Pretty tame day today:
- Made the Uploading documentation more specific, in that Contribtastic for Mac is the only client I know that supports Unified Uploader right now. There is upcoming support for others, do let me know if I've missed any that currently support it.
- Fixed a bug with uploadKeys being a dict instead of a list for EMK uploads.
- Improved validation to prevent issues with uploadKeys.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.25 04:27:00 -
[94] - Quote
If all continues to run quietly, I'll probably aim to tag "1.0" this weekend. I'd love some more eyes on the feed to make extra sure that everything is in order leading up to then. After that point, I'll be looking for people willing to volunteer additional gateways and/or relays. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Desmont McCallock
164
|
Posted - 2012.04.25 08:42:00 -
[95] - Quote
I build a script into the EMDR client to detect any bad formatted UF message. Until now every message comes in correct format. |

Desmont McCallock
165
|
Posted - 2012.04.25 15:33:00 -
[96] - Quote
https://forums.eveonline.com/default.aspx?g=posts&m=1190235#post1190235 https://forums.eveonline.com/default.aspx?g=posts&m=1190239#post1190239 |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.04.29 04:09:00 -
[97] - Quote
I'm moving houses this weekend/week, but snuck in an update to the EVE Central SMTP feeder to address occasional funkages, after I noticed we had lower inbound order data from them. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.01 18:56:00 -
[98] - Quote
The real-time map appears to be borked atm. We're aware of this, and I've harassed the author to fix :) EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.02 05:37:00 -
[99] - Quote
Tonight's updates:
- The real-time map has been un-borked: http://map.eve-emdr.com
- Worked with the EVE Central maintainer to correct their broken SMTP feed. We're getting some data, but it is currently sending out badly formed data (that we are filtering) at intervals. Talked to Yann about just using a direct ZeroMQ feed. Which leads to the next big development...
- I've got a prototype ZeroMQ gateway (IE: You can send orders to us over ZeroMQ instead of HTTP) running. Please PM me if you're interested. I'll have documentation soon.
- Further refinements to the uploader-visible error feedback.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.12 03:24:00 -
[100] - Quote
Recent changes:
- Added an experimental ZMQ gateway for sites that would like to relay market data more efficiently. Please PM me if you're interested. It's still preliminary, but I'd love to have help testing.
- Hopefully improved handling of gzipped uploads.
- Supporting the silly 'data' POST form-encoded requests.
- Lots of creature comforts with eventual machine volunteers in mind. All of the startup scripts are much improved.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Desmont McCallock
173
|
Posted - 2012.05.12 15:47:00 -
[101] - Quote
Code changes for post formed and compressed uploads are incorrect. Get in touch and I'll walk you through the fixes. |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.12 16:18:00 -
[102] - Quote
Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.
My rackspace stuff is all sourced out of the Chicago datacenter. |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.14 04:07:00 -
[103] - Quote
Barl Rathbone wrote:Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.
My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently. I'll definitely be in need of relays in the near future. Been dealing with moving into a new house unexpectedly, which slowed things down for a while, but I'm back in action now. Aiming to get one or two more issues worked through this week, then do a fial documentation push, then look to get some relays set up.
Rackspace or Linode are both just fine. It just can't be Atlanta+Linode, since that wouldn't give us any useful additional redundancy. Data usage from Atlanta to another Linode location may be free, though.
I'll be sending the call out for relays on the forums, and on the mailing list: https://groups.google.com/forum/#!forum/eve-emdr
Keep an eye on whichever one works best for you. The mailing list is super, super low volume, as I've mostly been doing things here. That'll probably be more for important announcements once we're up and running. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.14 04:11:00 -
[104] - Quote
I'm hoping we're reaching the end of the line as far as disruptive changes goes. I definitely need some eyes on things after this round of changes, as we have changed how we handle decompression and parsing.
- Decompression should now work properly (for real this time) for form-encoded POSTs. I'd strongly recommend against using form-encoded POSTs, as it's additional complexity, but the option is there for people with bad tools, or can't be bothered to read docs for their tools.
- Added more error handling for providing feedback when badly formed form-encoded POSTs are encountered.
EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Desmont McCallock
173
|
Posted - 2012.05.14 06:08:00 -
[105] - Quote
After running some testing:
- Method: POST (form-encoded) Compression: None Result: OK
- Method: POST (form-encoded) Compression: GZip Result: OK
- Method: POST (form-encoded) Compression: Deflate Result: OK
- Method: POST (entity) Compression: None Result: OK
- Method: POST (entity) Compression: GZip Result: OK
- Method: POST (entity) Compression: Deflate Result: OK
EMDR will be added in EMUU endpoints in few. |

Desmont McCallock
173
|
Posted - 2012.05.14 06:23:00 -
[106] - Quote
Now last thing that needs to be fixed is when uploading empty orders messages. I tried to setup EMDR locally (on Windows machine) but after a days headbanging I gave up. Didn't expected that setting up a Python client would be so hard. What really got on my nerves was the fact that I had to download, build, install all those external libraries a Python project is using and then finding out that it still doesn't run. |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.14 14:26:00 -
[107] - Quote
Desmont McCallock wrote:Now last thing that needs to be fixed is when uploading empty orders messages. I tried to setup EMDR locally (on Windows machine) but after a days headbanging I gave up. Didn't expected that setting up a Python client would be so hard. What really got on my nerves was the fact that I had to download, build, install all those external libraries a Python project is using and then finding out that it still doesn't run. Yeah, it can run on Windows, but considering that you've got a lot of extra work to do to install a compiler on Windows, it's probably not going to be something I want to officially support, even though it should be doable (in theory). As you saw, there are quite a few dependencies, gevent and libzmq being the ones needing a compiler.
I'll eventually write detailed instructions for Linux (which would be generic enough for the Mac OS users to follow), though. Once I've got the last few code rough spots fixed up, documentation will become my focus for the rest of the week. This will help make it possible to recruit people to run relays.
Something I have been looking into pyzmq-static:
http://pypi.python.org/pypi/pyzmq-static/
This is currently running an older version of pyzmq that isn't compatible with EMDR, but that bundles libzmq, which would reduce some of the annoyances you ran into. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Desmont McCallock
173
|
Posted - 2012.05.14 14:48:00 -
[108] - Quote
Actually it's zmq.green that's giving me the issues. It's not officially included in the distribution and I have to build, install it manually. Although I managed to do this at some time, still It through an exception on me with "Name _Socket not defined" in zmq\green\core.py |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.14 16:05:00 -
[109] - Quote
Ilyk Halibut wrote:Barl Rathbone wrote:Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.
My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently. I'll definitely be in need of relays in the near future. Been dealing with moving into a new house unexpectedly, which slowed things down for a while, but I'm back in action now. Aiming to get one or two more issues worked through this week, then do a fial documentation push, then look to get some relays set up. Rackspace or Linode are both just fine. It just can't be Atlanta+Linode, since that wouldn't give us any useful additional redundancy. Data usage from Atlanta to another Linode location may be free, though. I'll be sending the call out for relays on the forums, and on the mailing list: https://groups.google.com/forum/#!forum/eve-emdr Keep an eye on whichever one works best for you. The mailing list is super, super low volume, as I've mostly been doing things here. That'll probably be more for important announcements once we're up and running.
Nah - my stuff is in ORD for Rackspace and EWR (newark) for Linode so you'd have physical redundancy. |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.14 17:41:00 -
[110] - Quote
Desmont McCallock wrote:Actually it's zmq.green that's giving me the issues. It's not officially included in the distribution and I have to build, install it manually. Although I managed to do this at some time, still It through an exception on me with "Name _Socket not defined" in zmq\green\core.py
zmq.green is included as of pyzmq 2.2.0, make sure you upgrade to that. You'll need gevent as well. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Desmont McCallock
173
|
Posted - 2012.05.14 18:11:00 -
[111] - Quote
Unfortunately my friend, although it's in the repo, it's not included in the distributed msi or source zip files. Never the less I managed to get the source code from the repo, build it, install it but I can't make the EMDR source code to see the package. Now I get "ImportError: No module named green". Every other requirement is installed and working. |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.15 16:51:00 -
[112] - Quote
Desmont McCallock wrote:Unfortunately my friend, although it's in the repo, it's not included in the distributed msi or source zip files. Never the less I managed to get the source code from the repo, build it, install it but I can't make the EMDR source code to see the package. Now I get "ImportError: No module named green". Every other requirement is installed and working.
I'm in the same board with Desmont. I actually got the 2.2.0 egg built and installed but it broke pretty much everything in the process (e.g. now I'm getting undefined symbols on zmq_sendmsg when it tries to import the zmq.green lib or just zmq).
Frustrating to say the least. I'm trying to back everything out now and see if I can get my test harness working again before proceeding down the green path for a second try. |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.15 18:55:00 -
[113] - Quote
I think they're approaching a release, so hopefully it won't be as painful for much longer. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.15 19:25:00 -
[114] - Quote
Ilyk Halibut wrote:I think they're approaching a release, so hopefully it won't be as painful for much longer.
The problem is that you need the dev branch of 2.2.0, not the release tarball. If you clone it out of git it's fine, it includes the green files. If you use the release 2.2.0 tarball it doesn't contain green and (obviously) fails.
However, I'm not pulling data with worker threads. Next up, database insertion. |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.16 04:10:00 -
[115] - Quote
This will eventually be posted to the unified upload interchange format thread when it's in better shape, but I'm breaking all of the market data structures, along with serializers and de-serializers, out into their own project. Initially, this will just be for the unified format, but it'll be maintained in a manner to make it easy to drop others in if something new and better comes along.
https://github.com/gtaylor/EVE-Market-Data-Structures
There's going to be some pretty heavy lifting going on with this over the week, so I wouldn't use it for anything yet. It doesn't support the absence of orders/history for a given item+region ID combo yet, but that will be addressed shortly. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.16 13:16:00 -
[116] - Quote
Ilyk Halibut wrote:This will eventually be posted to the unified upload interchange format thread when it's in better shape, but I'm breaking all of the market data structures, along with serializers and de-serializers, out into their own project. Initially, this will just be for the unified format, but it'll be maintained in a manner to make it easy to drop others in if something new and better comes along. https://github.com/gtaylor/EVE-Market-Data-StructuresThere's going to be some pretty heavy lifting going on with this over the week, so I wouldn't use it for anything yet. It doesn't support the absence of orders/history for a given item+region ID combo yet, but that will be addressed shortly.
Nice! I was just working on my de-serializer but if you're going to publish one, hey why reinvent the wheel? :) |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
4
|
Posted - 2012.05.16 14:25:00 -
[117] - Quote
I'll probably be doing some pretty heavy re-working of what I have so far, because it made some bad assumptions. However, I'm hoping to have it in good shape and unit tested this week. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.16 20:19:00 -
[118] - Quote
Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.
Nice work Ilyk. Very nice work! |

Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
5
|
Posted - 2012.05.17 14:56:00 -
[119] - Quote
Barl Rathbone wrote:Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.
Nice work Ilyk. Very nice work!
Excellent, let me know if you see anything wrong. I'd like to have the data as scrutinized as humanly possible so we can catch anything before reaching "1.0". EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |

Barl Rathbone
Research Anonymous
0
|
Posted - 2012.05.17 16:56:00 -
[120] - Quote
Ilyk Halibut wrote:Barl Rathbone wrote:Well in a very quick and dirty method I've got a process running pulling the full feed and stuffing it into my database. Very quick, very dirty but I wanted to load some data so I'll let it run for a while.
Nice work Ilyk. Very nice work! Excellent, let me know if you see anything wrong. I'd like to have the data as scrutinized as humanly possible so we can catch anything before reaching "1.0".
It's a lot of data over time. My initial script to take in the data is pretty inefficient (straight up loop through the rows in each JSON message for inserts) and needs to be rewritten so it inserts all the rows in a JSON message at once rather than all the database connections being opened and closed.
So far the data base been pretty clean from what I've seen. I let the collector run for a few hours yesterday and didn't notice any problems that weren't on my side.
Let me know if you need another relay - I can have another linode or rackspace cloud box up and running quickly if needed. |
| |
|
| Pages: 1 2 3 [4] 5 6 7 8 9 :: one page |
| First page | Previous page | Next page | Last page |