Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6050
|
Posted - 2016.06.25 21:04:30 -
[1] - Quote
https://market.fuzzwork.co.uk/
May be of interest
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
John Volan
Royal Amarr Institute Amarr Empire
0
|
Posted - 2016.06.26 02:21:42 -
[2] - Quote
Very cool, thanks for doing this. Looking forward to the API. |
Tipa Riot
Federal Navy Academy Gallente Federation
2626
|
Posted - 2016.06.26 07:26:13 -
[3] - Quote
Wow, just wow! This is a giant leap in transparency, though a little frightening to see the details of my orders in public.
Thank you for this service!
EDIT: will CCP add this extra information also in-game? It's a bit odd you can get the details of any order in the market only via API, but not in-game.
I'm my own NPC alt.
|
Tipa Riot
Federal Navy Academy Gallente Federation
2626
|
Posted - 2016.06.26 07:46:48 -
[4] - Quote
What is the meaning of orderset? I see changes in this value even without a change in price or volume, sometimes 10 in a row at the same timestamp.
I'm my own NPC alt.
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6053
|
Posted - 2016.06.26 15:10:50 -
[5] - Quote
What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.
So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)
I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)
(an about page is coming. But I haven't written it yet)
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Zad Murrard
Frozen Dawn Inc Frozen Dawn Alliance
32
|
Posted - 2016.06.26 15:30:28 -
[6] - Quote
Very good. Looking forward to the API.
What is the intended role for the site in relation to existing market sites? |
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6053
|
Posted - 2016.06.26 15:43:43 -
[7] - Quote
Zad Murrard wrote:Very good. Looking forward to the API.
What is the intended role for the site in relation to existing market sites?
No intended role tbh. it's just another eve-central, or eve market data.
A pet project, for me to to stretch my analytics.
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Tipa Riot
Federal Navy Academy Gallente Federation
2626
|
Posted - 2016.06.26 17:03:30 -
[8] - Quote
Steve Ronuken wrote:What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.
So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)
I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)
(an about page is coming. But I haven't written it yet) Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course.
I'm my own NPC alt.
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6054
|
Posted - 2016.06.26 22:28:54 -
[9] - Quote
Tipa Riot wrote:Steve Ronuken wrote:What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.
So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)
I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)
(an about page is coming. But I haven't written it yet) Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course.
Storing only deltas complicates things, unfortunately. right now, I can store everything by just dumping it into the DB. Deltas would require a lookup per order being loaded (around 1.6 million of them)
It also means for display purposes, I'd need to pull everything ordered by the order set, then filter it down to the latest one.
I _may_ do the filtering on display of history, as that will be significantly smaller numbers being kicked around.
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Vaerah Vahrokha
Vahrokh Consulting
5716
|
Posted - 2016.06.27 18:40:53 -
[10] - Quote
Hello,
I am checking your nice website out.
I am trying to open this order.
However this triggers a bug:
Title Bad Connection to gateway
Text Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again
Auditing | Collateral holding and insurance | Consulting | PLEX for Good Charity
Twitter channel
|
|
TheSmokingHertog
Julia's Interstellar Trade Emperium
400
|
Posted - 2016.06.27 18:48:09 -
[11] - Quote
Steve Ronuken wrote:Tipa Riot wrote:Steve Ronuken wrote:What I'm doing is retrieving the order book once every 30 minutes, and storing it all with a different orderSet value.
So when you dig into an order's history, you're seeing it snapshots every 30 minutes. (I'm still deciding how long to keep them for. As it gets bigger, performance will suffer)
I'd retrieve more often, but processing the data takes a while. I could _probably_ bring it up to 15 minutes, but that would be very tight, and load could push it into overlapping (which could cascade)
(an about page is coming. But I haven't written it yet) Maybe this is possible, but you could just store an entry only if changed from previous snapshot (per order), otherwise discard. 30m is OK for analysis but 15m would be better of course. Storing only deltas complicates things, unfortunately. right now, I can store everything by just dumping it into the DB. Deltas would require a lookup per order being loaded (around 1.6 million of them) It also means for display purposes, I'd need to pull everything ordered by the order set, then filter it down to the latest one. I _may_ do the filtering on display of history, as that will be significantly smaller numbers being kicked around.
What if you would just start a delta on a lookup?
"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X
"Schrödinger's Missile. I dig it." ~ Makari Aeron
-= "Brain in a Box on Singularity" - April 2015 =-
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6062
|
Posted - 2016.06.28 16:51:07 -
[12] - Quote
Vaerah Vahrokha wrote:Hello, I am checking your nice website out. I am trying to open this order. However this triggers a bug: Title Bad Connection to gateway Text Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again
hrm. For some reason that page is taking too long to generate. (it's probably to do with the face I'm inserting and deleting around 81 million records per day. The db doesn't like it) One of the reasons I'm considering replacing the server with one with SSDs instead. The load stage is pegging the disk for a fairly variable period of time.
If you were to look _right now_, it works.
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Vaerah Vahrokha
Vahrokh Consulting
5716
|
Posted - 2016.07.01 07:21:46 -
[13] - Quote
Steve Ronuken wrote:Vaerah Vahrokha wrote:Hello, I am checking your nice website out. I am trying to open this order. However this triggers a bug: Title Bad Connection to gateway Text Looks like something is wrong with the database. Steve will need to kick it repeatedly until it works again hrm. For some reason that page is taking too long to generate. (it's probably to do with the face I'm inserting and deleting around 81 million records per day. The db doesn't like it) One of the reasons I'm considering replacing the server with one with SSDs instead. The load stage is pegging the disk for a fairly variable period of time. If you were to look _right now_, it works. Edit: Digging into this, yes, it's down to the market load pegging the disk. SSDs _should_ fix this, but my options are either waiting for one company to replenish their stocks ( soyoustart ) or paying about double what I'm paying right now for that server (moving to hetzner. It's a significantly better server than the soyostart one (500GB vs 300GB for storage, 32GB vs 16GB for ram), but there's also a one off setup fee.) While I'm not looking to make money on any of this stuff (because I'm not insane , and it's not why I do it), that's going to mean pushing it to next month. (Unless I get a run on my Patreon. Somehow, I doubt that'll happen )
I stumbled upon this kind of issues so many times.
I fixed them by writing my own query (result set) caching classes, usually working with memchached / REDIS. With some creativity you can even implement "cache priming" during idle time, so you don't get those huge data access peaks.
Some other times (I see you have some good RAM) I got helped by directly putting most used pages on Nginx + Varnish.
Hope some of this stuff helps.
Auditing | Collateral holding and insurance | Consulting | PLEX for Good Charity
Twitter channel
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6083
|
Posted - 2016.07.04 12:42:55 -
[14] - Quote
https://market.fuzzwork.co.uk/api/ is now up and running.
Still need to write the function for googledocs users
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
James Zealot
ALEHEADS Apocalypse Now.
22
|
Posted - 2016.07.05 05:02:09 -
[15] - Quote
I love you for this :-) |
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |