Pages: 1 2 [3] 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 24 post(s) |
Luscius Uta
109
|
Posted - 2014.10.19 19:52:00 -
[61] - Quote
CCP DeNormalized wrote:virm pasuul wrote:CCP DeNormalized wrote:virm pasuul wrote:I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day. It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?" I am considering turning the users off because they are spoiling my database. we discuss similar things :) An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :) Make everything on the market a fixed price, then the table will become much smaller. Seems a good compromise to me :) In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it... So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :) I got over ruled on that as well (i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)
The only reason why EVE needs achievements
People hate highsec for various reasons. Mine is the terrible metallic music that plays on and on. |
|
CCP DeNormalized
C C P C C P Alliance
104
|
Posted - 2014.10.19 22:04:00 -
[62] - Quote
Jandice Ymladris wrote: I wonder how much data our assets are occupying on the server? As unlike other MMO's, we can have stuff everywhere indefinatly, so I suspect it would be a big drain on data storage? Or not really as it's more static?
the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc...
every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :)
CCP DeNormalized DBA Virtual World Operations |
|
Tzeentchful Nakrar
Dixon Cox Butte Preservation Society Black Legion.
0
|
Posted - 2014.10.20 01:59:00 -
[63] - Quote
CCP DeNormalized wrote:Jandice Ymladris wrote: I wonder how much data our assets are occupying on the server? As unlike other MMO's, we can have stuff everywhere indefinatly, so I suspect it would be a big drain on data storage? Or not really as it's more static?
the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc... every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :) That is a lot of rows. You guys must have some really insane servers to serve that amount of data at the seed you do. I do have a quick question. What is the oldest item a players has? |
Bienator II
madmen of the skies
2906
|
Posted - 2014.10.20 03:59:00 -
[64] - Quote
i am wondering if the DB would be still useable when every item in eve would be unstacked. Since if it would ... can we pretty please have manufacturer info and other metadata for items? ;) eve style bounties (done) dust boarding parties imagine there is war and everybody cloaks - join FW |
Max Kolonko
High Voltage Industries Ash Alliance
471
|
Posted - 2014.10.20 05:56:00 -
[65] - Quote
Bienator II wrote:i am wondering if the DB would be still useable when every item in eve would be unstacked. Since if it would ... can we pretty please have manufacturer info and other metadata for items? ;)
that trit Read and support: Don't mess with OUR WH's What is Your stance on WH stuff? |
WhyYouHeffToBeMad IsOnlyGame
Republic Military School Minmatar Republic
214
|
Posted - 2014.10.20 08:27:00 -
[66] - Quote
Sakurako Kimino wrote:CCP DeNormalized wrote:Toriessian wrote:Jandice Ymladris wrote:CCP DeNormalized wrote:it is a sell order, that started with 20 units - 17 sold and 3 left :)
it's only like 1.6 mil per unit and not the highest in the region. Must be something very rarely traded then! Or perhaps the creator just likes to see how long he can keep running it... Still, darn impressive lasting that long. Its about the right price for synth boosters. Perhaps, but this isn't those!! :) if you tell us the item we might buy them out lol that is why he doesn't want to tell which item it is so to not ruin the EVE record for the longest ongoing sell order. A.K.A Hodor Von Grootenberg |
Varisto
North Star Science And Industry
4
|
Posted - 2014.10.20 08:34:00 -
[67] - Quote
So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data... |
|
CCP DeNormalized
C C P C C P Alliance
114
|
Posted - 2014.10.20 10:28:00 -
[68] - Quote
Varisto wrote:So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data...
I've asked our SAN guys that very question in the past :)
They say upwards of 5 year life span depending on how many writes, etc... keep in mind these are enterprise grade SSD's (with the insane prices to boot) and have extra 'stuff' to keep them alive longer :)
100% moar stuff!!! yeaaa!!!! hehe
We've yet to lose one, but I believe we're getting close to end of life and may likely replace in the coming year or so.
CCP DeNormalized DBA Virtual World Operations |
|
|
CCP DeNormalized
C C P C C P Alliance
114
|
Posted - 2014.10.20 10:40:00 -
[69] - Quote
Tzeentchful Nakrar wrote: That is a lot of rows. You guys must have some really insane servers to serve that amount of data at the seed you do. I do have a quick question. What is the oldest item a players has?
The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed.
That much RAM coupled with fast SSD's (and properly indexed data) and you can do quite a bit :)
In terms of oldest item, it's a very difficult thing to find. Up until 4 years ago or so, we were recycling ID's (32 bit) so we are really able to say beyond that.
Now we use 64 bit and can trace how old items are, but I think it's fair to say that there are items in hangars from the beginning :) CCP DeNormalized DBA Virtual World Operations |
|
Johan Civire
The Lyran Empire
867
|
Posted - 2014.10.20 11:09:00 -
[70] - Quote
This is good reading stuff :D i like it. But about that order can we have a name or what do he or she sells or is that forbidden? i like to know now ? :D |
|
Jandice Ymladris
Aurora Arcology
974
|
Posted - 2014.10.20 11:10:00 -
[71] - Quote
I'm surprised the inventory 'only' takes 259 GB of space, considering we tend to litter stuff everywhere as our characters age (I admit I'm terrible at it, think my asset list expands over a hundred stations....) So sounds like a pretty efficient system you got (I guess it helps that you don't need to 'generate' a history for the items as well, at least the ones you can stack)
On the other hand 2.2 billion rows, and one row per item/stack per location, that's one humongous list.... Albeit I presume this includes inactive accounts as well.
That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing? The result of the YC116 Sleeper data research race: Tech 3 Destroyers! A summary of the ancient civilizations of New Eden |
Max Kolonko
High Voltage Industries Ash Alliance
474
|
Posted - 2014.10.20 11:21:00 -
[72] - Quote
Jandice Ymladris wrote:
That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing?
Rams going into hundreds or thousands of gigs for dedicated database clusters are slowly becoming standard for large databases. I think google was one of the first to utilize it in their hadoop design.
Read and support: Don't mess with OUR WH's What is Your stance on WH stuff? |
Hengle Teron
I'm So Meta Even This Acronym
2591
|
Posted - 2014.10.20 11:22:00 -
[73] - Quote
CCP DeNormalized wrote:
The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed. :)
Mind blown |
Arline Kley
PIE Inc. Praetoria Imperialis Excubitoris
411
|
Posted - 2014.10.20 11:25:00 -
[74] - Quote
Jandice Ymladris wrote:That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing?
512GB of RAM is pretty big; the cluster I am running here is something like 120GB RAM for 9 seperate servers (each virtualized) whilst the data is sat on a 8TB SAN running tradtional HDD's. We did try to get SSD's but sadly the costs were prohibitive.
I would give my left arm to work in CCP's datacentre. Blessed are those that carry the Empress' Light; with it they destroy the shadows |
|
CCP DeNormalized
C C P C C P Alliance
119
|
Posted - 2014.10.20 11:25:00 -
[75] - Quote
Johan Civire wrote:This is good reading stuff :D i like it. But about that order can we have a name or what do he or she sells or is that forbidden? i like to know now ? :D
I wouldn't tell, no... community may choose to do something with this info, but if not it'll just be one more mystery of the universe :) CCP DeNormalized DBA Virtual World Operations |
|
Kousaka Otsu Shigure
9
|
Posted - 2014.10.20 11:26:00 -
[76] - Quote
One of my side projects is creating an out-of-game contacts list. Is our ingame contacts list cap somehow related to how the database works?
I'm at ~24k hisec contacts at the moment and i must admit, im such a noob with databases. |
|
CCP DeNormalized
C C P C C P Alliance
119
|
Posted - 2014.10.20 11:43:00 -
[77] - Quote
Kousaka Otsu Shigure wrote:One of my side projects is creating an out-of-game contacts list. Is our ingame contacts list cap somehow related to how the database works?
I'm at ~24k hisec contacts at the moment and i must admit, im such a noob with databases.
In most cases any sort of cap would be done for performance reasons, either due to the DB or the client itself. I can't see the DB being an issue here however.
Mind you, if you were to have 24,000 contacts in-game, i would think the overhead of loading those in the client and doing whatever happens with those, would definitely have a negative impact on client side performance :)
CCP DeNormalized DBA Virtual World Operations |
|
Wacktopia
Noir. Suddenly Spaceships.
711
|
Posted - 2014.10.20 12:46:00 -
[78] - Quote
Chribba wrote:CCP DeNormalized wrote:DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total. So that's why you're not letting me read mails further back than a month Stop purging and give me my mails back!
See, all the cool kids are deleting their data. :) Kitchen sink? Seriousy, get your ship together - -áFleet-Up.com |
Arcos Vandymion
White Beast Inc.
65
|
Posted - 2014.10.20 14:25:00 -
[79] - Quote
Jandice Ymladris wrote:I'm surprised the inventory 'only' takes 259 GB of space, considering we tend to litter stuff everywhere as our characters age (I admit I'm terrible at it, think my asset list expands over a hundred stations....) So sounds like a pretty efficient system you got (I guess it helps that you don't need to 'generate' a history for the items as well, at least the ones you can stack)
I perferr my cchaos concentrated into one abysmal eye of terror... I did have to rent freighter services a handful of times allready to ferry stuff between stations.
Currently everything is on one station and some assests in null that i can't get sold ...
Regarding the contacts - I recently did a single mission for the majority of corporations in EVE and whenever I open my standings tab now (which really could need nifty little arrows to expand or collapse the tree) it needs a while. Can't imagine what the game would do if you had a lsit of 24k to pull and display. |
Gospadin
School of Applied Knowledge Caldari State
227
|
Posted - 2014.10.20 17:29:00 -
[80] - Quote
Varisto wrote:So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data...
There are essentially no actual field usage models that will burn out the latest generation of enterprise SSDs with 10DWD or above ratings. 10 DWD to an 800GB drive is 100MB/s non-stop for 5 years. In a normal 80/20 read/write workload typical of many databases, essentially the drive will run bus-saturated for the entire warranty duration.
Most people don't understand what a petabyte of data written to a single device looks like, while drives launched a year ago are good for multiple petabytes per drive. Build an array out of them, and you're talking multiple tens of petabytes. That rating accounts for all internal data movement as well, meaning those petabytes are true user petabytes.
|
|
5yndr0m3
Origin. Black Legion.
31
|
Posted - 2014.10.20 19:19:00 -
[81] - Quote
ISD LackOfFaith wrote:[b]Disclaimer: what follows is a nerd's best guess,
Hope this helps!
Proving once again ISD dont read the threads before they post/lock them.
|
5yndr0m3
Origin. Black Legion.
31
|
Posted - 2014.10.20 19:19:25 -
[82] - Quote
ISD LackOfFaith wrote:[b]Disclaimer: what follows is a nerd's best guess,
Hope this helps!
Proving once again ISD dont read the threads before they post/lock them.
|
Arline Kley
PIE Inc. Praetoria Imperialis Excubitoris
412
|
Posted - 2014.10.20 22:15:00 -
[83] - Quote
Jandice Ymladris wrote:that feels story.
The Program yawned into awareness, wondering how long it had been asleep for. A small flashing icon appeared on its screen, which it discounted immediately.
"Damn messages.. " It started the usual rituals it had gotten used to for the last few years: check eve-mail, wallet then market. Open browser, head to the forums. Observe. Repeat.
The blinking appeared again within seconds and in frustration, the program opened the message.
All items sold. Market order closed.
It blinked, re-reading the message over and over again. Was it true? Had it actually been fulfilled after all these years? Immediately, it checked the market; it didn't appear there. It was gone. For the first time in years, the program smiled, leaning back into its chair and letting out a long sigh, closing its eyes in relief.
"Your time, Program, has come."
The voice made the Program sit up violently, turning to the source of it. Stood in the centre of the room was a solitary individual, pale of tone, gaunt and clad in a well fitting black suit and edged in a pale blue light, its eyes a solid black save for emerald green irises.
"Who are you? How did you get in here?" the Program asked, a mild tone of fear and confusion in its voice.
"Your time has come Program. We are the Administrators and We have been observing you for some time." Its voice was monotone, and straight to the point.
"We?" thought the Program, but upon hearing the word Administrators, its cores froze. What had it done? It had heard that the Administrators where the rule makers of the Great Game. Many programs had fallen to the edicts of the Administrators, some exiled into the void while others reappeared under new forms and guises after debating their continued existence. Their rule was harsh, but fair.
"But I have followed your edicts... altered my code when requested. Why have I been targeted for term....?" the Program's voice was panicky, reaching out for the chair arm to stand, only for it to phase through. it leapt upright, only to see its static framework, leant down in a position of sleep.
"Your coding has been observed operating for 7 years, Program. We have run the Scripts and Indexed the Tables. Your coding eluded our attempts to correct it. However, We allowed it to continue. Now that your coding is complete, We are offering you peace in the Binary Halls, your Program to added to the Grand Library. You merely need to follow Us towards it."
The Program was still looking at its framework, seeing it slowly dissipating into the ether, block by block. Eventually the chair was empty, the only evidence of its existence being the screen, which it too, was fading to black, awaiting its next incumbent.
The Administrator stood in silence, awaiting the Program's decision. What felt like an eternity was merely a few ticks on the universal Clock, before the Program nodded, and followed the Administrator towards its final resting place.
===
Been thinking about this all day.. Blessed are those that carry the Empress' Light; with it they destroy the shadows |
Arline Kley
PIE Inc. Praetoria Imperialis Excubitoris
424
|
Posted - 2014.10.20 22:15:55 -
[84] - Quote
Jandice Ymladris wrote:that feels story.
The Program yawned into awareness, wondering how long it had been asleep for. A small flashing icon appeared on its screen, which it discounted immediately.
"Damn messages.. " It started the usual rituals it had gotten used to for the last few years: check eve-mail, wallet then market. Open browser, head to the forums. Observe. Repeat.
The blinking appeared again within seconds and in frustration, the program opened the message.
All items sold. Market order closed.
It blinked, re-reading the message over and over again. Was it true? Had it actually been fulfilled after all these years? Immediately, it checked the market; it didn't appear there. It was gone. For the first time in years, the program smiled, leaning back into its chair and letting out a long sigh, closing its eyes in relief.
"Your time, Program, has come."
The voice made the Program sit up violently, turning to the source of it. Stood in the centre of the room was a solitary individual, pale of tone, gaunt and clad in a well fitting black suit and edged in a pale blue light, its eyes a solid black save for emerald green irises.
"Who are you? How did you get in here?" the Program asked, a mild tone of fear and confusion in its voice.
"Your time has come Program. We are the Administrators and We have been observing you for some time." Its voice was monotone, and straight to the point.
"We?" thought the Program, but upon hearing the word Administrators, its cores froze. What had it done? It had heard that the Administrators where the rule makers of the Great Game. Many programs had fallen to the edicts of the Administrators, some exiled into the void while others reappeared under new forms and guises after debating their continued existence. Their rule was harsh, but fair.
"But I have followed your edicts... altered my code when requested. Why have I been targeted for term....?" the Program's voice was panicky, reaching out for the chair arm to stand, only for it to phase through. it leapt upright, only to see its static framework, leant down in a position of sleep.
"Your coding has been observed operating for 7 years, Program. We have run the Scripts and Indexed the Tables. Your coding eluded our attempts to correct it. However, We allowed it to continue. Now that your coding is complete, We are offering you peace in the Binary Halls, your Program to added to the Grand Library. You merely need to follow Us towards it."
The Program was still looking at its framework, seeing it slowly dissipating into the ether, block by block. Eventually the chair was empty, the only evidence of its existence being the screen, which it too, was fading to black, awaiting its next incumbent.
The Administrator stood in silence, awaiting the Program's decision. What felt like an eternity was merely a few ticks on the universal Clock, before the Program nodded, and followed the Administrator towards its final resting place.
===
Been thinking about this all day..
Blessed are those that carry the Empress' Light; with it they destroy the shadows
|
Amose Amanum
The Oasis Group TOG - The Older Gamers Alliance
0
|
Posted - 2014.11.17 04:09:30 -
[85] - Quote
CCP DeNormalized wrote:
In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it...
So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha
There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :)
I got over ruled on that as well
(i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)
How many are left now? |
Moraguth
Ranger Corp Vae. Victis.
124
|
Posted - 2014.11.17 10:32:35 -
[86] - Quote
I remember way back in the day, around the time that you moved the servers maybe, we got an update saying that TQ had reached 1 ton. How heavy is our darling baby now?
OH! Can we get stats (size and weight) for Sisi too? Is she a sever of 1 box on a rack somewhere all by her lonesome or what? Or does she just run as a dedicated node that responds to a different IP address?
I got a Feature Added!
Stop calling an Abaddon "abba-dawn". It is "uh-bad-in"
dictionary.com/abaddon
|
Amose Amanum
The Oasis Group TOG - The Older Gamers Alliance
0
|
Posted - 2014.11.17 12:09:51 -
[87] - Quote
Oh, in case anybody is interested, I have located and purchased what I believe to be one of the remaining three items. I am confident in my knowledge and awaiting confirmation from the seller.
I will be interested to see who is next to solve this. The evidence is all out in public. |
virm pasuul
Mine 'N' Refine Yulai Federation
177
|
Posted - 2014.11.17 16:15:57 -
[88] - Quote
I don't know if DeNormalized would be allowed to confirm or deny that you bought the right thing. One sure thing to do would be to buy the remaining two, given the cost is next to nothing. If DeNormalized and his fellow OCDers at CCP track you down and shower you in beer at fanfest then assume you got it right :)
|
45thtiger 0109
AL3XAND3R.
113
|
Posted - 2014.11.17 22:22:41 -
[89] - Quote
This is a great read & best thread I have ever seen for a long time and great response from CCP
Well done to the OP for this thread
**You Have to take the good with the bad
and the bad with the good.
Welcome to EvE OnLiNe**
|
Agondray
Avenger Mercenaries VOID Intergalactic Forces
155
|
Posted - 2014.11.17 22:45:48 -
[90] - Quote
HHD space is over rated and only needed when the computers power down and boot up, then it shifts everything over onto RAM for quick access *hence why when you get a computer and it states the OS needs 6GB of RAM and your computer comes with 8GB RAM, that 6GB is what the OS needs to run leaving you with 2GB to run other programs but will only pull those programs from the drive when queued for the information and then sends it to the RAM for operation*
so in short when EVE starts up the data moves from the drives to RAM and occasionally refreshes data *IE your microsoft word autosave or you clicking the save button sends that document data to your HDD and not the RAMso if power loss happens it doesn't vanish.
"Sarcasm is the Recourse of a weak mind." -Dr. Smith
|
|
|
|
|
Pages: 1 2 [3] 4 :: one page |
First page | Previous page | Next page | Last page |