| Pages: [1] 2 :: one page |
|
|
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
HUGO DRAX
   |
Posted - 2005.08.07 14:59:00 -
[1]
I am just curious how many servers do they need to run for this game and what kind of connection do they use to the world?? ie multiple T1s etc.. and is eve 100% inhouse code etc..
just curious on how the whole thing works.
|
Xelios
|
Posted - 2005.08.07 15:08:00 -
[2]
Blog 1
Blog 2
Some info to get you started, some of it is a bit out of date though..
____________________________________________________________________
|
HUGO DRAX
|
Posted - 2005.08.08 01:27:00 -
[3]
humm. so it looks like they have 12 front end servers, probably 2 nics on the proxies one nic facing the outside world and the other nic probably to a switch facing the SQL servers and each SQL server connects via fiberchannel to a mini san. I guess they probably use a midrange cisco router. Wow looks like everything would fit in 1 rack. The lower power/operations + NAP rental costs after buying the hardware and the initial software R&D being paid off means that EvE could run for a long time with 60K customers and make a decent profit if expenses are kept low. It would be nice to see some pics of the whole deal though.
|
Nac MacFeegle
|
Posted - 2005.08.08 02:42:00 -
[4]
Multiple T1s? How... 1990s. 
For their link to the internet, it's most likely a rate-limited OC-3... preferably two, redundant to seperate carriers. Well, that's what I'd do, anyway... I come from an e-commerce background where downtime == death.
More likely is that they're buying the bandwidth from their colo in the UK, and they're the ones with the fat net pipes (again, preferably redundant).
All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance. |
SpaceDrake Storyteller
|
Posted - 2005.08.08 04:13:00 -
[5]
Actually, reading those I get the impression of two racks, or at least one rack divided into two vertical columns. You've got the proxies in the first column/rack, the Cisco routers in between (only three for the entire cluster? FFS, hope that's been updated), and then the game SQL servers.
Still fairly low-maint, though, with the exception of the SQL servers since they get backed up every night. Even then, running one server still has economic advantages.
--------------
What good are actions if there's no one to tell the tale afterward?
Player of the character "Lucca Deradi." Former player of "Andre Ricard." |
Froggy
|
Posted - 2005.08.08 11:58:00 -
[6]
Edited by: Froggy on 08/08/2005 11:59:10
Current count is:
14 proxy server
55 Sol servers
2 SQL servers
Disk array takes about 1/2 rack
then there is allso around 10 misc. servers
All this currently takes up 5 racks and there isnt much free space :)
Singularity test cluster has an additional 2 racks but there is alot of free space there.
And yes TQ cluster is on a multihomed connection provided by our hosting partner.
in 3-6 monts these numbers will prob. be out dated.
|
Gigi Barbagrigia
|
Posted - 2005.08.08 12:16:00 -
[7]
10 misc servers? Bloody hell, you're running UT server aren't you. Invite only? :)
-----
42 |
TheJay
|
Posted - 2005.08.08 12:31:00 -
[8]
It's where Oveur stores his ****! 
|
Del Narveux
|
Posted - 2005.08.08 12:38:00 -
[9]
Edited by: Del Narveux on 08/08/2005 12:39:13
I always figured at leats one of them was TomB's House of Warez. 
Anyway, could one of you dev types be a cool dude and say about how much disk space all the items, ships, characters, etc. take up? This aspect of MMOG operation has always fascinated me, perhaps its the thought you would need an ungodly amount of data compression/streamlining to keep it at a managable figure.
_________________
[SAK] And Proud Of It!
aka Cpt Bogus
Is that my torped sig cloaking your base? |
Kahn Moquil
|
Posted - 2005.08.08 12:43:00 -
[10]
 Originally by: Del Narveux
Edited by: Del Narveux on 08/08/2005 12:39:13
I always figured at leats one of them was TomB's House of Warez.
Anyway, could one of you dev types be a cool dude and say about how much disk space all the items, ships, characters, etc. take up? This aspect of MMOG operation has always fascinated me, perhaps its the thought you would need an ungodly amount of data compression/streamlining to keep it at a managable figure.
A disk array that takes about half a rack. I'm guessing somewhere around 750 Gb, maybe less if the individual disks aren't of the biggest size available.
|
|
|
Skarsnik
|
Posted - 2005.08.08 12:47:00 -
[11]
Originally by: Froggy
Edited by: Froggy on 08/08/2005 11:59:10
Current count is:
14 proxy server
55 Sol servers
2 SQL servers
Disk array takes about 1/2 rack
then there is allso around 10 misc. servers
All this currently takes up 5 racks and there isnt much free space :)
Singularity test cluster has an additional 2 racks but there is alot of free space there.
And yes TQ cluster is on a multihomed connection provided by our hosting partner.
in 3-6 monts these numbers will prob. be out dated.
[geek]
I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering?
Only reason is the SQL back end will be getting a massive hammering and would like to know if it's Active/Passive or Active/Active setup..
And why not upgrade the SQL boxes to Win2k3 from 2k and then MSSQL2k5 and use DB mirroring as the failover.. just a suggestion of course 
[/geek]
---------------------------------
No Slugs were harmed in the creation of this signature
--------------------------------- |
Sylie Silentsong
|
Posted - 2005.08.08 13:03:00 -
[12]
Flat text files 4tw! 
|
FireFoxx80
|
Posted - 2005.08.08 13:10:00 -
[13]
This is geek **** isn't it?
Possibly one of the 23 # ex: P-TMC | USAC |
Skarsnik
|
Posted - 2005.08.08 15:35:00 -
[14]
Originally by: FireFoxx80
This is geek **** isn't it?
Yeah sorry for my Geek Outburst - I just get wood when Clusters are mentioned
But your right it's all geek, being geek can be good for you sometimes - although I forget why
---------------------------------
No Slugs were harmed in the creation of this signature
--------------------------------- |
Hllaxiu
|
Posted - 2005.08.08 15:38:00 -
[15]
Originally by: Skarsnik
I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering?
I thought the servers were something freeer. :P
Proud member of Elite Academy. |
Froggy
|
Posted - 2005.08.08 16:36:00 -
[16]
Ill try and sum up some of the questions here.
DB size is about 180GB but it isnÆt really the size thatÆs the problem rather the immense IO load on the db hence all the disks.
SQL cluster runs on w2k3 enterprise server with MS Active/standby cluster.
We have done some testing with w2k3 server on the proxy and sol layers of the test cluster and maybe we will upgrade the main cluster to that in the future but currently no decision has been made.
More fun facts about the Cluster:
EVE has about 150cpuÆs and 200GB memory. The cluster has more then doubled in size since launch probably closer to triple the original power.
We have some very geeky upgrades planed for the cluster but Im not shore how much of that I can reveal so Im going to save that for an ultra geeky blog later this year.
|
HUGO DRAX
|
Posted - 2005.08.08 16:58:00 -
[17]
Originally by: Hllaxiu
Originally by: Skarsnik
I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering?
I thought the servers were something freeer. :P
hehe, yeah me too. and a penguin as the mascot.
|
Doc Brown
|
Posted - 2005.08.08 17:21:00 -
[18]
Originally by: Froggy
We have some very geeky upgrades planed for the cluster but Im not shore how much of that I can reveal so Im going to save that for an ultra geeky blog later this year.
Please do blog those when done, and don't skimp on the details as some of us do understand them all.
Question: Do do any loadbalancing (via hardware loadbalancer..ie: F5, etc) in the NOC?
Also, I know it's possible (in theory) to use the hardware loadbalancers to distribute SQL but I don't know how effective that really is.. have you guys looked into that?
_________________________________________________
There are no bad ideas, only bad implementations. |
Froggy
|
Posted - 2005.08.08 19:54:00 -
[19]
Edited by: Froggy on 08/08/2005 19:55:43
Yes we use Nortel/Alteon HW loadbalancers to distribute traffic on the proxy servers.
And no we havent looked into useing HW loadbalancers for the SQL cluster since the issue there is more of IO then cpu power
|
Nac MacFeegle
|
Posted - 2005.08.08 21:29:00 -
[20]
12,000 active connections, two SQL servers... yeah, that could be an I/O problem right there... :)
I'm wondering if, at this point, its more of a problem with bandwidth saturation on the SQL servers' motherboards. A PCI bus can only take so much... a northbridge can only take so much.
While it may be technically correct, I just don't see two servers as being a 'cluster'. :) I'd see about form of clustering where data is distributed across multiple servers with their own disk... keep down I/O choking.
All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance. |
|
|
CmdrRat
|
Posted - 2005.08.08 21:55:00 -
[21]
Originally by: FireFoxx80
This is geek **** isn't it?
Yes, and I've got wood.
Anyway, I'm supprised to see that the Database is less than 200GB, I would have figured with all those bookmarks and such it would have grown really large. Assuming of course that each of the bookmarks needs its own entry in the DB.
_
____
_______
_________________________________________________________
_________________________________________________________
|
Grimpak
|
Posted - 2005.08.08 22:05:00 -
[22]
I read the whole thread and I had a brain fart
me no comprende all this geeky stuff
-------------------
Quote:
Fragm's Oversized Ego Cannon barely scratches the forums, inflicting omgnoonecares damage
|
Capsicum
|
Posted - 2005.08.08 22:42:00 -
[23]
I'm certainly impressed :)
Thanks for the info!
{The Forum Rules}|{Email Us!} |
SengH
|
Posted - 2005.08.08 22:53:00 -
[24]
More importantly... how many servers power the forums?
Joking aside, is CCP ever going to make the move to solid state fibre channel drives for the database? Wouldn't that solve the i/o problems overnight? Although that would be horrendously expensive.....
|
Lyra VX
|
Posted - 2005.08.08 23:02:00 -
[25]
I feel unusually ungeeked by asking this - what are Sol servers? Unless we're referring to Solaris I don't think I've ever heard of this
|
Nac MacFeegle
|
Posted - 2005.08.08 23:26:00 -
[26]
Originally by: Lyra VX
I feel unusually ungeeked by asking this - what are Sol servers? Unless we're referring to Solaris I don't think I've ever heard of this
I suspect they're the servers where the star systems 'live' when people are in them.
All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance. |
Captin Biltmore
|
Posted - 2005.08.08 23:27:00 -
[27]
I'm really supprized you answered all those questions.....I think I would be more concerned about security than appeasing the whims of a few people's interests. Especially since you have 60,000+ credit card numbers stored on your servers.
Assasin For Hire - Contact in game |
Winterblink
|
Posted - 2005.08.08 23:29:00 -
[28]
Originally by: Nac MacFeegle
12,000 active connections, two SQL servers... yeah, that could be an I/O problem right there... :)
I'm wondering if, at this point, its more of a problem with bandwidth saturation on the SQL servers' motherboards. A PCI bus can only take so much... a northbridge can only take so much.
While it may be technically correct, I just don't see two servers as being a 'cluster'. :) I'd see about form of clustering where data is distributed across multiple servers with their own disk... keep down I/O choking.
This is why the system is a multi-tier one, so there aren't 12000 connections happening on the SQL servers at the same time. The beauty of having business tiers is that they buffer the database a lot. Not all the game transactions necessitate a database hit to accomplish the task.
SQL Server actually isn't as bad as most people make it out to be. If you know what you're doing you can tune it extremely well, and in some cases do things more efficiently than with an Oracle database.
___winterblink/warp_drive_active/eve_nature_vraie// |
Drilla
|
Posted - 2005.08.08 23:47:00 -
[29]
Solid-state drives for the SQL logs helped the E10000K Sun servers we run for db backend at Maersk. We had massive IO issues on the Oracle installations and by using solid-state disks for the SQL logs almost nullified the IO issues in one upgrade.
Only downside was the pricetag on the disks but hey - Maersk isnt excatly poor when flipping out for 8 E10000K/E15000K servers at a time - all fully stocked ofc. 
EVE System Security - Killboard (still early alpha) |
Nac MacFeegle
|
Posted - 2005.08.09 01:21:00 -
[30]
Originally by: Winterblink
This is why the system is a multi-tier one, so there aren't 12000 connections happening on the SQL servers at the same time. The beauty of having business tiers is that they buffer the database a lot. Not all the game transactions necessitate a database hit to accomplish the task.
Its still a hell of a lot of data to be asking two servers to read/write, even if they are beefy.
Quote:
SQL Server actually isn't as bad as most people make it out to be. If you know what you're doing you can tune it extremely well, and in some cases do things more efficiently than with an Oracle database.
I've come to a conclusion lately: "All databases suck. Some just suck more expensively than others." Oracle's expensive and has a design and interface that still harkens to the days of mainframes, but at least it runs on various platforms. Microsoft zaps you once for the SQL Server licenses, zaps you again on the Windows Server licenses, and then if you want to have a system capable of running handling a ton of traffic you have to pony up for some beefy x86 servers... a platform that started life as a desktop computer and got shoehorned into the server niche...
All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance. |
|
|
|
|
| |
|
| Pages: [1] 2 :: one page |
| First page | Previous page | Next page | Last page |