Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
11
|
Posted - 2012.12.12 04:52:00 -
[1] - Quote
Now that we've been successfully running for a while, I wanted to see if I could gather a list of projects and sites using EMDR. The results of this will be posted here:
http://www.eve-emdr.com/en/latest/sites.html
Either reply here or PM me directly with your publicly available sites and projects that use EMDR. A name, a URL, and a brief description of what you're doing will suffice.
Thanks! EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Squizz Caphinator
Greater Order Of Destruction Happy Endings
56
|
Posted - 2012.12.12 14:39:00 -
[2] - Quote
You probably won't get much response unless you turned off EMDR and noted who complained :)
Great service tho, keep up the good work! http://evewho.com - Alliance and Corporation Member Listings http://evechatter.com - Free Alliance and Corporation forums for all. |
Steve Ronuken
Fuzzwork Enterprises
848
|
Posted - 2012.12.12 15:43:00 -
[3] - Quote
http://www.fuzzworks.co.uk/
Most of my projects here are using market data directly derived from the feed. A few aren't, but that's mostly because I need to fiddle with my consumer to grab more than The Forge's data. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
11
|
Posted - 2012.12.12 19:56:00 -
[4] - Quote
Squizz Caphinator wrote:You probably won't get much response unless you turned off EMDR and noted who complained :)
Great service tho, keep up the good work!
Hah, you are probably very right. I tossed this up on the mailing list too, hoping for more replies there. We're serving something like 30+ steady, 24/7 connections now, so I'm hoping to at least hear what some of those are. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
11
|
Posted - 2012.12.12 19:57:00 -
[5] - Quote
Steve Ronuken wrote:http://www.fuzzworks.co.uk/
Most of my projects here are using market data directly derived from the feed. A few aren't, but that's mostly because I need to fiddle with my consumer to grab more than The Forge's data.
Do you have any public links to any of these projects? I'd like to link them directly so they can see. EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Ilyk Halibut
Blackwater USA Inc. Against ALL Authorities
11
|
Posted - 2012.12.14 15:08:00 -
[6] - Quote
Well, we've got 4 out of the 30+ connections accounted for: http://www.eve-emdr.com/en/latest/sites.html EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com |
Steve Ronuken
Fuzzwork Enterprises
851
|
Posted - 2012.12.14 15:25:00 -
[7] - Quote
Ilyk Halibut wrote:Steve Ronuken wrote:http://www.fuzzworks.co.uk/
Most of my projects here are using market data directly derived from the feed. A few aren't, but that's mostly because I need to fiddle with my consumer to grab more than The Forge's data. Do you have any public links to any of these projects? I'd like to link them directly so they can see.
http://www.fuzzwork.co.uk/blueprints/ grabs all the pricing data from a 5% buy/sell of the forge.
I'll be converting http://www.fuzzwork.co.uk/ore/ over at some point
http://www.fuzzwork.co.uk/market/buyprices.csv http://www.fuzzwork.co.uk/market/buyprices.json http://www.fuzzwork.co.uk/market/prices.csv http://www.fuzzwork.co.uk/market/prices.json
Are dumps of the data kept (restricted to stuff useful for making stuff) fro the forge, taken once hourly. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
Bosence
Frog Morton Industries Anuran Origin
4
|
Posted - 2012.12.16 00:15:00 -
[8] - Quote
I might be one connection at the moment, for a project in development. Just so you can account for one, but it's not that stable and it's not anyway near production use. |
Ydnari
Estrale Frontiers Project Wildfire
137
|
Posted - 2012.12.16 01:35:00 -
[9] - Quote
I'm using EMDR and am probably one of your 30, I use it to get prices and price history for an internal corp management system and industrial planner, and also use the same data to monitor my market orders vs. orders in other regions, and look for trade opportunities and trends. Currently it's not externally accessible, I do keep meaning to plan on cleaning it up for wider usage but to be honest it's a fair way off that. |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
131
|
Posted - 2012.12.18 10:48:00 -
[10] - Quote
EVE Marketeer data gathering is currently also running strong again.
I did note that the US Central relay seems to have blown up though, so I had to switch relays.
http://www.evemarketeer.com
It may be some time before you see much more than the placeholder page though. Developer/Creator of EVE Marketeer
|
|
Desmont McCallock
230
|
Posted - 2012.12.18 11:17:00 -
[11] - Quote
Callean Drevus wrote:EVE Marketeer data gathering is currently also running strong again. I did note that the US Central relay seems to have blown up though, so I had to switch relays. http://www.evemarketeer.comIt may be some time before you see much more than the placeholder page though. And why do you use the US relay? There are several EU relays you can use (knowing that your project runs from EU). |
Callean Drevus
Icosahedron Crafts and Shipping Silent Infinity
131
|
Posted - 2012.12.18 11:32:00 -
[12] - Quote
I've moved around both in terms of servers and my own location.
However, after thinking about it for a bit, my server was always in the US, so I would've used it either way :) Developer/Creator of EVE Marketeer
|
Desmont McCallock
230
|
Posted - 2012.12.18 12:21:00 -
[13] - Quote
Callean Drevus wrote:I've moved around both in terms of servers and my own location.
However, after thinking about it for a bit, my server was always in the US, so I would've used it either way :) In that case, OK. |
Badmoons
Industrial Nightmare Inc
0
|
Posted - 2012.12.19 16:32:00 -
[14] - Quote
Has anyone else noticed some clients sending dates for the duration column instead of numbers? |
Badmoons
Industrial Nightmare Inc
0
|
Posted - 2012.12.19 16:41:00 -
[15] - Quote
*sigh* nevermind, found my problem. |
Conquistador Alvarez
Center for Advanced Studies Gallente Federation
0
|
Posted - 2012.12.27 16:02:00 -
[16] - Quote
Hi there, i'm trying to get this running in C# but the code that i found on the EMDR Site doenst work at all
I am using visual studio 2012, any ideas or helps out there?
it starts with all "using ZMQ" is not working and i have to use "using ZeroMQ" and diffrent bindings |
Desmont McCallock
263
|
Posted - 2012.12.27 16:40:00 -
[17] - Quote
I'll have a look when I get home. |
Quantix BlackStar
The Praxis Initiative Gentlemen's Agreement
5
|
Posted - 2012.12.27 17:12:00 -
[18] - Quote
Conquistador Alvarez wrote:Hi there, i'm trying to get this running in C# but the code that i found on the EMDR Site doenst work at all
I am using visual studio 2012, any ideas or helps out there?
it starts with all "using ZMQ" is not working and i have to use "using ZeroMQ" and diffrent bindings
Yes, the EMDR documentation is out of date to the current version of ZeroMQ. I wrote a basic EMDR subscriber in VS2012 using C# this morning, and if you'd like a copy to use as an example send me a pm with an email addy I can send it to (I'm just heading out the door atm, so I didn't upload it to the net yet).
Cheers,
Q EveHQ is a multi-faceted character application for EVE Online. Includes pilot monitoring, skill queue planning, ship fitting, industry and more. Get the latest version at http://evehq.net
|
Desmont McCallock
263
|
Posted - 2012.12.27 17:52:00 -
[19] - Quote
Code still works fine with clrzmq2.2.5 (https://github.com/zeromq/clrzmq/downloads). It's just that the example subscribes to a dead endpoint (us-central). |
Conquistador Alvarez
Center for Advanced Studies Gallente Federation
0
|
Posted - 2012.12.27 18:23:00 -
[20] - Quote
thx all we figured out where the problem is, first it was with the new release and second we made something wrong with one dll.
now we have to figure out what we are going to do with the datas |
|
Desmont McCallock
276
|
Posted - 2012.12.27 18:27:00 -
[21] - Quote
Glad you sort it out. Still you owe me those CSPA I payed for EVE mailing you (just kidding, don't dare send any ISK). |
Quantix BlackStar
The Praxis Initiative Gentlemen's Agreement
6
|
Posted - 2012.12.27 22:33:00 -
[22] - Quote
Desmont McCallock wrote:Code still works fine with clrzmq.2.2.5 (https://github.com/zeromq/clrzmq/downloads).
@Quantix BlackStar The code doesn't use ZeroMQ installer but the .NET 'clr' binding (http://www.zeromq.org/bindings:clr). So don't rush on judgement.
There were some other changes I made in order to get this to work (I'm using the ZMQ nuget distribution)... I'll diff the two and follow up with you privately if they were significant.
Either way, the EveHQ project will be moving to utilize this data (consuming via eve-central), and also pushing data back up.
Cheers,
Q EveHQ is a multi-faceted character application for EVE Online. Includes pilot monitoring, skill queue planning, ship fitting, industry and more. Get the latest version at http://evehq.net
|
EveDataRules
World Domination Inc Templis Dragonaors
0
|
Posted - 2012.12.28 03:17:00 -
[23] - Quote
Using it: http://www.evedata.org
Much better than the mail push and http pulls. It would be nice if there were some more examples of pushing the data into a SQL server without crippling it though.
I am currently using shared structures to deal with duplicates, nuking the old data if it was reported later than the new data and using a single thread to post updates to MySQL every 120 seconds while taking advantage of MySQL's insert/update multiple rows feature. Loving it.
Anyone have any other suggestions on pushing data to a database? |
Steve Ronuken
Fuzzwork Enterprises
988
|
Posted - 2012.12.28 10:01:00 -
[24] - Quote
Depends what data you're storing, really.
I'm using memcache as my primary store as the data I'm interested in tends to be thrown around quickly. Takes around an hour to populate the majority of it. Hourly I throw the data down onto disk in csv and json files. I don't care about the history though. Just percentiles for buying and selling.
If I was wanting something more persistent, I'd probably use memcache as a primary store, and have a process in the background that assembles the inserts and batches them every few minutes.
I suspect the other way would be to use one of the other message queues to hold the 'good' data, if you're wanting a FIFO processing of updates.
so:
EMDR Consumer: zeromq for accepting data from EMDR memcache for checking the state you have held (to eliminate repeats and old data) ActiveMQ/RabbitMQ/Qpid for storing the good non-duplicate data
Your own personal consumer: client for the broker above the database. Runs on a slower cycle for batching the changes. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
Muscaat
EVE Markets
26
|
Posted - 2012.12.29 15:34:00 -
[25] - Quote
I'm working on a new data-collection backend for EVE Markets using EMDR. No ETA yet on when it might go live though |
Steve Ronuken
Fuzzwork Enterprises
1005
|
Posted - 2012.12.29 16:17:00 -
[26] - Quote
EveDataRules wrote:Using it: http://www.evedata.orgMuch better than the mail push and http pulls. It would be nice if there were some more examples of pushing the data into a SQL server without crippling it though. I am currently using shared structures to deal with duplicates, nuking the old data if it was reported later than the new data and using a single thread to post updates to MySQL every 120 seconds while taking advantage of MySQL's insert/update multiple rows feature. Loving it. Anyone have any other suggestions on pushing data to a database?
Oh, I was meaning to add:
Make sure you're using innodb, rather than myisam for the table you're updating. It handles the locking better for updates. myisam is very bad for that. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
EveDataRules
World Domination Inc
0
|
Posted - 2012.12.29 19:41:00 -
[27] - Quote
I am using InnoDB, it is very nice, the market update and history update are one single query each, not sure if it will still lock per row or do the whole table in that instance. It is very fast though, around 12k inserts a second on old hardware (3ghz P4, 2gb DDR2 Ram, velociraptor).
I have been thinking about rewriting my consumer into C to make it a little faster and handle memory better. Currently it is written in Perl.
I am not sure about putting another application in the middle of this; is ActiveMQ/RabbitMQ/Qpid for creating a store then writing said store to disk for querying later? Maybe using LOADFILE in MySQL with the dump would be faster?
Right now i am just using shared hashes of arrays in Perl with a key of orderID and something a bit more complicated for the history. |
Steve Ronuken
Fuzzwork Enterprises
1005
|
Posted - 2012.12.29 19:59:00 -
[28] - Quote
EveDataRules wrote:I am using InnoDB, it is very nice, the market update and history update are one single query each, not sure if it will still lock per row or do the whole table in that instance. It is very fast though, around 12k inserts a second on old hardware (3ghz P4, 2gb DDR2 Ram, velociraptor).
I have been thinking about rewriting my consumer into C to make it a little faster and handle memory better. Currently it is written in Perl.
I am not sure about putting another application in the middle of this; is ActiveMQ/RabbitMQ/Qpid for creating a store then writing said store to disk for querying later? Maybe using LOADFILE in MySQL with the dump would be faster?
Right now i am just using shared hashes of arrays in Perl with a key of orderID and something a bit more complicated for the history.
The inbetween queue is to store the list of 'what needs to be updated'. So it can be handled in a batched fashion. Disk is an option though. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |