Pages: [1] 2 3 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 24 post(s) |
virm pasuul
Mine 'N' Refine Yulai Federation
162
|
Posted - 2014.10.17 13:07:00 -
[1] - Quote
Just being curious. How big in Gb is Tranquility please? I'm guessing big :) |
Doc Fury
Furious Enterprises
6846
|
Posted - 2014.10.17 13:10:00 -
[2] - Quote
It's as big as a baby Ren!
The accumulated filth of all their sex and murder will foam up about their waists and all the ho's and politicians will look up and shout 'Save us!' and I'll look down, and whisper 'Hodor'. |
Ramona McCandless
The McCandless Clan
7025
|
Posted - 2014.10.17 13:11:00 -
[3] - Quote
Its not as big as a Texan Bar "Many have joined the battle, many have survived the tests and trials, but countless have fallen because they weren't the sharpest, the fastest thinking, the most devious, the most ruthless or most intelligent. -áLog in and Compete!"-á- CCP Falcon
|
Hengle Teron
I'm So Meta Even This Acronym
2587
|
Posted - 2014.10.17 14:19:00 -
[4] - Quote
Can you tell me that in cubic meters? please ! |
Erin Crawford
307
|
Posted - 2014.10.17 14:29:00 -
[5] - Quote
as big as a Fedo's bottom! |
Guttripper
State War Academy Caldari State
556
|
Posted - 2014.10.17 15:45:00 -
[6] - Quote
Definitely smaller than most male teenager's pornography collection... |
The Forum Alt
Hedion University Amarr Empire
42
|
Posted - 2014.10.17 16:08:00 -
[7] - Quote
I have it on good authority that the size of the database is smaller than 1000 petabytes but bigger than 1 byte. |
|
Chribba
Otherworld Enterprises Otherworld Empire
13027
|
Posted - 2014.10.17 16:20:00 -
[8] - Quote
Last time I heard it was a few TB, that was some years ago, so with Jita's daily market transactions I would guess 10x the size now.
/c
|
|
Unsuccessful At Everything
The Troll Bridge
17147
|
Posted - 2014.10.17 16:23:00 -
[9] - Quote
Im sure the database is the same size in Great Britain as it would be anywhere else in the world.
Since the cessation of their usefulness is imminent, may I appropriate your belongings? |
|
CCP DeNormalized
C C P C C P Alliance
18
|
Posted - 2014.10.17 16:24:00 -
[10] - Quote
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total. CCP DeNormalized DBA Virtual World Operations |
|
|
|
Chribba
Otherworld Enterprises Otherworld Empire
13028
|
Posted - 2014.10.17 16:26:00 -
[11] - Quote
CCP DeNormalized wrote:DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total. So that's why you're not letting me read mails further back than a month Stop purging and give me my mails back!
|
|
|
CCP DeNormalized
C C P C C P Alliance
20
|
Posted - 2014.10.17 16:28:00 -
[12] - Quote
Chribba wrote:Last time I heard it was a few TB, that was some years ago, so with Jita's daily market transactions I would guess 10x the size now.
/c
Over the course of the last year, the market tables have grown by about 500 GB :) CCP DeNormalized DBA Virtual World Operations |
|
|
ISD LackOfFaith
ISD Community Communications Liaisons
1757
|
Posted - 2014.10.17 16:47:00 -
[13] - Quote
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.
TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.
A MySQL dump of the static database (item types, solar systems, orbital objects, etc) is around 300 MB.
Total so far: 300 MB
The Community Toolkit downloads for Types, Renders, and Icons are an extra 600 MB or so, zipped. Unzipped at a conservative 50% compression rate, that's 1.2 GB of data just for various static images.
Total so far: 1.5 GB
While we're assuming all static data is also part of the database, we can add the size of a regular install too; say, 10 GB.
Total so far: 11.5 GB
Somewhat less dynamic are character portraits, each of which are saved in varying sizes, from 1024x1024 JPGs down to 32x32 JPG icons. Based on my own portrait, i'm going to estimate the total per-character of these is 250 KB. In 2013, Eve reached "over 500,000 subscribers". Assuming an average of 2.5 characters per account (not everyone uses all 3 slots, but many do), that's 1,250,000 characters. That means 312.5 GB in just character portraits.
Total so far: 324 GB
Of course, that's active subscriptions. How many trial players (or player alts) started accounts that are now inactive? Eve is a pretty rough niche game, so let's say there are 1,500,000 of those. That's another 375 GB of portraits.
Total so far: 699 GB
Now, let's look at ships. The 10 year commemoration video estimates 40,000,000 ships lost. Let's add another 5,000,000 for a violent year. Assuming the number of ships lost per day has been going up linearly since the game's launch (it hasn't, but let's assume), that means that after the (roughly) 4200 days of Eve's lifespan, we're getting about 21,000 lost ships per day. Every time you lose a ship, how many more do you have sitting around? I have no access to those stats, and no idea where to get them, but I'm going to ballpark 10 for the average. 10 ships per 1,250,000 active character is 12,500,000 ships. Add one noobship for every inactive account and you get 14,000,000 ships.
Each ship has a bunch (20-30) of attributes associated with it. Each of those is a 64-bit (8 byte) integer value. 25 attributes per ship, with 8 bytes per attribute and 14,000,000 ships, that's 2.8 GB of data.
Total so far: 702 GB (note: I have only considered assembled ships so far)
But many of those ships have modules fit to them! Let's estimate 8 modules/rigs fit per ship. Each one of those also has a bunch of attributes, and blah blah blah... another 1 GB of stuff.
Total so far: 703 GB
What about other items? Well... Eve-central has a 200 MB gzipped market dump for every day. I'm not going to bother downloading and unzipping those, and call it 1 GB per day. If that's the data size of just the items being traded, how much stuff is there in game? How many buy/sell orders does the average character have going at any one time, compared to their items? Well, no more than 5, unless they have trading skills trained (few non-dedicated traders do). Let's just call it 5 orders. Eve players are pack rats and keep lots of stuff scattered all around the universe. I just checked and it looks like I have many thousands of things scattered about. Though, compensating for alts and new players, I'm going to ballpark 2,000 items (stacks of things) average per character. That means that on average, players have 500 item more items than they do market orders. If an item takes roughly the same amount of data to store, then that's 500 GB of data about peoples' stuff. Plus 1 GB of the actual current orders.
Total so far: 1,304 GB
What about stateful stuff in space? NPCs? Drones? Wrecks? Corpses? Large collidable objects? Asteroids? Mission structures? Wormholes? POSes? Player outposts? Mobile depots? Bubbles? This number is extremely hard to gauge because many of these aren't simple items. They all have positional data, and probably a variety of extra stuff stored server-side to make them work. Their number also varies wildly from system to system. I'm not sure how to approach it, but I'm going to ballpark that it takes multiple times the space it takes to store peoples' stuff (which is relatively static and boring). Let's call it just under 3x, so 2,000 GB.
Total so far: 3,304 GB
You could add logs and other non-critical data to make it an even 3.5 TB.
I can't think of anything else that needs storing, and at this point I've done so much estimation that the answer may be off by quite a lot.
Still... 3.5 TB is not that much, is it? I mean, on Newegg, a 2 TB solid state drive costs just $3300. For about $6000-7000 you could store all of Eve's data on a very fast drive, right?
Well... not so much. Eve is a big place, and a single SSD has no way to keep up with how intensely it would have to be used if it served as the central repository. Also, can you imagine what would happen if the drive broke? You would need extensive replication of all that data both for performance and for backup purposes. On top of that, remember that Tranquility is distributed across 60 (?) nodes, each with a local copy of high-demand data (your ships and stuff in space) plus other cached data.
Considering all the duplication of data necessary to have a system as robust as TQ (yeah, don't laugh, TQ is amazingly robust for what it is), I would estimate there to be something on the order of 100 TB of data floating around in storage between all the machines in TQ put together. Maybe a bit less. Maybe a bit more. Who knows, I don't work there.
Hope this helps! ISD LackOfFaith Captain Community Communication Liaisons (CCLs) Interstellar Services Department I do not respond to Eve Mail or anything other than the forums. |
|
TheMercenaryKing
StarFleet Enterprises Intrepid Crossing
294
|
Posted - 2014.10.17 16:47:00 -
[14] - Quote
CCP DeNormalized wrote:DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.
How often do you backup the servers? Daily via deduplication?
I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives? 15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization. |
Bienator II
madmen of the skies
2898
|
Posted - 2014.10.17 16:49:00 -
[15] - Quote
* stacked his loot to make the DB smaller eve style bounties (done) dust boarding parties imagine there is war and everybody cloaks - join FW |
|
CCP DeNormalized
C C P C C P Alliance
21
|
Posted - 2014.10.17 16:59:00 -
[16] - Quote
ISD LackOfFaith wrote:Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.
TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.
Great effort LoF!
As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.
the inventory system takes up about 250 GB (a single table w/ 2-3 indexes) the market system takes up around 300 GB (2-3 tables and multiple indexes) in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))
CCP DeNormalized DBA Virtual World Operations |
|
|
CCP DeNormalized
C C P C C P Alliance
22
|
Posted - 2014.10.17 17:08:00 -
[17] - Quote
TheMercenaryKing wrote:
How often do you backup the servers? Daily via deduplication?
I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives? 15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.
DB is split across 2 tiers of storage, SSD (enterprise level intel disks in v7000 enclosures) for the heavily accessed stuff and 15k SAS disks for the second 'slower' tier.
I'm not that knowledgeable abuot the SAN side and how the block sizes are handed there, but as the LUN's are given to us, we setup our volumes @ 64K block size as this is the optimal allocation for SQL (64k = 1 extent = 8x8KB pages)
We do 2 full backups per day (written across 8 files for disk melting IO dumps) which take a little over 1 hour each... Hits the disks for around 400 MB/s
CCP DeNormalized DBA Virtual World Operations |
|
MiliasColds
The Elite Few Inc. The Methodical Alliance
21
|
Posted - 2014.10.17 17:09:00 -
[18] - Quote
it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P
edi: seems i was wrong and beaten to the punch by 1s |
virm pasuul
Mine 'N' Refine Yulai Federation
164
|
Posted - 2014.10.17 17:09:00 -
[19] - Quote
TY DeNormalized, and thank you for caring for our sandpit. |
|
CCP DeNormalized
C C P C C P Alliance
22
|
Posted - 2014.10.17 17:12:00 -
[20] - Quote
MiliasColds wrote:it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P
edi: seems i was wrong and beaten to the punch by 1s
We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.
We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).
CCP DeNormalized DBA Virtual World Operations |
|
|
MiliasColds
The Elite Few Inc. The Methodical Alliance
21
|
Posted - 2014.10.17 17:18:00 -
[21] - Quote
CCP DeNormalized wrote:MiliasColds wrote:it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P
edi: seems i was wrong and beaten to the punch by 1s We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult. We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).
least i wasn't just horribly misinformed, simply out of date :), but dat 1M randon IOPS /s :P |
virm pasuul
Mine 'N' Refine Yulai Federation
164
|
Posted - 2014.10.17 17:18:00 -
[22] - Quote
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day. It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?" I am considering turning the users off because they are spoiling my database. |
Hengle Teron
I'm So Meta Even This Acronym
2587
|
Posted - 2014.10.17 17:25:00 -
[23] - Quote
CCP DeNormalized wrote:ISD LackOfFaith wrote:Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.
TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.
Great effort LoF! As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB. the inventory system takes up about 250 GB (a single table w/ 2-3 indexes) the market system takes up around 300 GB (2-3 tables and multiple indexes) in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :)) ok, but how much is that in cubic meters... ? |
Adrie Atticus
Shadows of Rebellion The Bastion
439
|
Posted - 2014.10.17 17:38:00 -
[24] - Quote
Hengle Teron wrote:CCP DeNormalized wrote:ISD LackOfFaith wrote:Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.
TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.
Great effort LoF! As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB. the inventory system takes up about 250 GB (a single table w/ 2-3 indexes) the market system takes up around 300 GB (2-3 tables and multiple indexes) in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :)) ok, but how much is that in cubic meters... ?
0.02 m3 per module, not sure how many, but I'd guess it's a full rack; 1.43 m3 per full 42U rack. |
|
ISD LackOfFaith
ISD Community Communications Liaisons
1760
|
Posted - 2014.10.17 17:40:00 -
[25] - Quote
CCP DeNormalized wrote:ISD LackOfFaith wrote:Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.
TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.
Great effort LoF! As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB. the inventory system takes up about 250 GB (a single table w/ 2-3 indexes) the market system takes up around 300 GB (2-3 tables and multiple indexes) in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :)) Woo! Hooray ballparking!
Also, I completely forgot about the mail system, oops. ISD LackOfFaith Captain Community Communication Liaisons (CCLs) Interstellar Services Department I do not respond to Eve Mail or anything other than the forums. |
|
|
CCP DeNormalized
C C P C C P Alliance
26
|
Posted - 2014.10.17 17:44:00 -
[26] - Quote
virm pasuul wrote:I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day. It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?" I am considering turning the users off because they are spoiling my database.
we discuss similar things :)
An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :) CCP DeNormalized DBA Virtual World Operations |
|
Chainsaw Plankton
IDLE GUNS IDLE EMPIRE
708
|
Posted - 2014.10.17 17:46:00 -
[27] - Quote
CCP DeNormalized wrote:MiliasColds wrote:it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P
edi: seems i was wrong and beaten to the punch by 1s We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult. We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).
I remember seeing something about the Ramsan in a devblog a while back. all i know is thing sounded shiny I'll join the chorus asking CCP, don't take my fancy names away from me! In the name of the Limos, the Malkuth, and the Arbalest, so help me pod - Mara Rinn |
virm pasuul
Mine 'N' Refine Yulai Federation
165
|
Posted - 2014.10.17 17:55:00 -
[28] - Quote
CCP DeNormalized wrote:virm pasuul wrote:I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day. It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?" I am considering turning the users off because they are spoiling my database. we discuss similar things :) An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :)
Make everything on the market a fixed price, then the table will become much smaller. Seems a good compromise to me :) |
Caviar Liberta
Moira. Villore Accords
722
|
Posted - 2014.10.17 17:57:00 -
[29] - Quote
TheMercenaryKing wrote:CCP DeNormalized wrote:DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total. How often do you backup the servers? Daily via deduplication? I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives? 15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.
The intial back up would be the most time consuming. Setting up the back to only copy any files that has been changed or added would make the task a lot easier then. |
|
CCP DeNormalized
C C P C C P Alliance
32
|
Posted - 2014.10.17 18:02:00 -
[30] - Quote
Caviar Liberta wrote:TheMercenaryKing wrote:CCP DeNormalized wrote:DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total. How often do you backup the servers? Daily via deduplication? I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives? 15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization. The intial back up would be the most time consuming. Setting up the back to only copy any files that has been changed or added would make the task a lot easier then.
we do different things w/ each backup - the secondary one gets shipped to a different server for reporting purposes, but you are correct, a differential would make sense in most cases here CCP DeNormalized DBA Virtual World Operations |
|
|
|
|
|
Pages: [1] 2 3 4 :: one page |
First page | Previous page | Next page | Last page |