EVE Search - EvE servers, technical info??

Check My IP Information

All Channels

EVE General Discussion

EvE servers, technical info??

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 :: [one page]

Author	Thread Statistics \| Show CCP posts - 0 post(s)
HUGO DRAX	Posted - 2005.08.07 14:59:00 - [1] I am just curious how many servers do they need to run for this game and what kind of connection do they use to the world?? ie multiple T1s etc.. and is eve 100% inhouse code etc.. just curious on how the whole thing works.
Xelios	Posted - 2005.08.07 15:08:00 - [2] Blog 1 Blog 2 Some info to get you started, some of it is a bit out of date though.. ____________________________________________________________________
HUGO DRAX	Posted - 2005.08.08 01:27:00 - [3] humm. so it looks like they have 12 front end servers, probably 2 nics on the proxies one nic facing the outside world and the other nic probably to a switch facing the SQL servers and each SQL server connects via fiberchannel to a mini san. I guess they probably use a midrange cisco router. Wow looks like everything would fit in 1 rack. The lower power/operations + NAP rental costs after buying the hardware and the initial software R&D being paid off means that EvE could run for a long time with 60K customers and make a decent profit if expenses are kept low. It would be nice to see some pics of the whole deal though.
Nac MacFeegle	Posted - 2005.08.08 02:42:00 - [4] Multiple T1s? How... 1990s. For their link to the internet, it's most likely a rate-limited OC-3... preferably two, redundant to seperate carriers. Well, that's what I'd do, anyway... I come from an e-commerce background where downtime == death. More likely is that they're buying the bandwidth from their colo in the UK, and they're the ones with the fat net pipes (again, preferably redundant). All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance.
SpaceDrake Storyteller	Posted - 2005.08.08 04:13:00 - [5] Actually, reading those I get the impression of two racks, or at least one rack divided into two vertical columns. You've got the proxies in the first column/rack, the Cisco routers in between (only three for the entire cluster? FFS, hope that's been updated), and then the game SQL servers. Still fairly low-maint, though, with the exception of the SQL servers since they get backed up every night. Even then, running one server still has economic advantages. -------------- What good are actions if there's no one to tell the tale afterward? Player of the character "Lucca Deradi." Former player of "Andre Ricard."
Froggy	Posted - 2005.08.08 11:58:00 - [6] Edited by: Froggy on 08/08/2005 11:59:10 Current count is: 14 proxy server 55 Sol servers 2 SQL servers Disk array takes about 1/2 rack then there is allso around 10 misc. servers All this currently takes up 5 racks and there isnt much free space :) Singularity test cluster has an additional 2 racks but there is alot of free space there. And yes TQ cluster is on a multihomed connection provided by our hosting partner. in 3-6 monts these numbers will prob. be out dated.
Gigi Barbagrigia	Posted - 2005.08.08 12:16:00 - [7] 10 misc servers? Bloody hell, you're running UT server aren't you. Invite only? :) ----- 42
TheJay	Posted - 2005.08.08 12:31:00 - [8] It's where Oveur stores his ****!
Del Narveux	Posted - 2005.08.08 12:38:00 - [9] Edited by: Del Narveux on 08/08/2005 12:39:13 I always figured at leats one of them was TomB's House of Warez. Anyway, could one of you dev types be a cool dude and say about how much disk space all the items, ships, characters, etc. take up? This aspect of MMOG operation has always fascinated me, perhaps its the thought you would need an ungodly amount of data compression/streamlining to keep it at a managable figure. _________________ [SAK] And Proud Of It! aka Cpt Bogus Is that my torped sig cloaking your base?
Kahn Moquil	Posted - 2005.08.08 12:43:00 - [10] Originally by: Del Narveux Edited by: Del Narveux on 08/08/2005 12:39:13 I always figured at leats one of them was TomB's House of Warez. Anyway, could one of you dev types be a cool dude and say about how much disk space all the items, ships, characters, etc. take up? This aspect of MMOG operation has always fascinated me, perhaps its the thought you would need an ungodly amount of data compression/streamlining to keep it at a managable figure. A disk array that takes about half a rack. I'm guessing somewhere around 750 Gb, maybe less if the individual disks aren't of the biggest size available.

Skarsnik	Posted - 2005.08.08 12:47:00 - [11] Originally by: Froggy Edited by: Froggy on 08/08/2005 11:59:10 Current count is: 14 proxy server 55 Sol servers 2 SQL servers Disk array takes about 1/2 rack then there is allso around 10 misc. servers All this currently takes up 5 racks and there isnt much free space :) Singularity test cluster has an additional 2 racks but there is alot of free space there. And yes TQ cluster is on a multihomed connection provided by our hosting partner. in 3-6 monts these numbers will prob. be out dated. [geek] I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering? Only reason is the SQL back end will be getting a massive hammering and would like to know if it's Active/Passive or Active/Active setup.. And why not upgrade the SQL boxes to Win2k3 from 2k and then MSSQL2k5 and use DB mirroring as the failover.. just a suggestion of course [/geek] --------------------------------- No Slugs were harmed in the creation of this signature ---------------------------------
Sylie Silentsong	Posted - 2005.08.08 13:03:00 - [12] Flat text files 4tw!
FireFoxx80	Posted - 2005.08.08 13:10:00 - [13] This is geek **** isn't it? Possibly one of the 23 # ex: P-TMC \| USAC
Skarsnik	Posted - 2005.08.08 15:35:00 - [14] Originally by: FireFoxx80 This is geek **** isn't it? Yeah sorry for my Geek Outburst - I just get wood when Clusters are mentioned But your right it's all geek, being geek can be good for you sometimes - although I forget why --------------------------------- No Slugs were harmed in the creation of this signature ---------------------------------
Hllaxiu	Posted - 2005.08.08 15:38:00 - [15] Originally by: Skarsnik I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering? I thought the servers were something freeer. :P Proud member of Elite Academy.
Froggy	Posted - 2005.08.08 16:36:00 - [16] Ill try and sum up some of the questions here. DB size is about 180GB but it isnÆt really the size thatÆs the problem rather the immense IO load on the db hence all the disks. SQL cluster runs on w2k3 enterprise server with MS Active/standby cluster. We have done some testing with w2k3 server on the proxy and sol layers of the test cluster and maybe we will upgrade the main cluster to that in the future but currently no decision has been made. More fun facts about the Cluster: EVE has about 150cpuÆs and 200GB memory. The cluster has more then doubled in size since launch probably closer to triple the original power. We have some very geeky upgrades planed for the cluster but Im not shore how much of that I can reveal so Im going to save that for an ultra geeky blog later this year.
HUGO DRAX	Posted - 2005.08.08 16:58:00 - [17] Originally by: Hllaxiu Originally by: Skarsnik I'd like to know what clustering your using on the SQL boxes tbh, is it Base Microsofts clustering effort or something a little more substantiated like Veritas Clustering? I thought the servers were something freeer. :P hehe, yeah me too. and a penguin as the mascot.
Doc Brown	Posted - 2005.08.08 17:21:00 - [18] Originally by: Froggy We have some very geeky upgrades planed for the cluster but Im not shore how much of that I can reveal so Im going to save that for an ultra geeky blog later this year. Please do blog those when done, and don't skimp on the details as some of us do understand them all. Question: Do do any loadbalancing (via hardware loadbalancer..ie: F5, etc) in the NOC? Also, I know it's possible (in theory) to use the hardware loadbalancers to distribute SQL but I don't know how effective that really is.. have you guys looked into that? _________________________________________________ There are no bad ideas, only bad implementations.
Froggy	Posted - 2005.08.08 19:54:00 - [19] Edited by: Froggy on 08/08/2005 19:55:43 Yes we use Nortel/Alteon HW loadbalancers to distribute traffic on the proxy servers. And no we havent looked into useing HW loadbalancers for the SQL cluster since the issue there is more of IO then cpu power
Nac MacFeegle	Posted - 2005.08.08 21:29:00 - [20] 12,000 active connections, two SQL servers... yeah, that could be an I/O problem right there... :) I'm wondering if, at this point, its more of a problem with bandwidth saturation on the SQL servers' motherboards. A PCI bus can only take so much... a northbridge can only take so much. While it may be technically correct, I just don't see two servers as being a 'cluster'. :) I'd see about form of clustering where data is distributed across multiple servers with their own disk... keep down I/O choking. All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance.

CmdrRat	Posted - 2005.08.08 21:55:00 - [21] Originally by: FireFoxx80 This is geek **** isn't it? Yes, and I've got wood. Anyway, I'm supprised to see that the Database is less than 200GB, I would have figured with all those bookmarks and such it would have grown really large. Assuming of course that each of the bookmarks needs its own entry in the DB. _ ____ _______ _________________________________________________________ _________________________________________________________
Grimpak	Posted - 2005.08.08 22:05:00 - [22] I read the whole thread and I had a brain fart me no comprende all this geeky stuff ------------------- Quote: Fragm's Oversized Ego Cannon barely scratches the forums, inflicting omgnoonecares damage
Capsicum	Posted - 2005.08.08 22:42:00 - [23] I'm certainly impressed :) Thanks for the info! {The Forum Rules}\|{Email Us!}
SengH	Posted - 2005.08.08 22:53:00 - [24] More importantly... how many servers power the forums? Joking aside, is CCP ever going to make the move to solid state fibre channel drives for the database? Wouldn't that solve the i/o problems overnight? Although that would be horrendously expensive.....
Lyra VX	Posted - 2005.08.08 23:02:00 - [25] I feel unusually ungeeked by asking this - what are Sol servers? Unless we're referring to Solaris I don't think I've ever heard of this
Nac MacFeegle	Posted - 2005.08.08 23:26:00 - [26] Originally by: Lyra VX I feel unusually ungeeked by asking this - what are Sol servers? Unless we're referring to Solaris I don't think I've ever heard of this I suspect they're the servers where the star systems 'live' when people are in them. All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance.
Captin Biltmore	Posted - 2005.08.08 23:27:00 - [27] I'm really supprized you answered all those questions.....I think I would be more concerned about security than appeasing the whims of a few people's interests. Especially since you have 60,000+ credit card numbers stored on your servers. Assasin For Hire - Contact in game
Winterblink	Posted - 2005.08.08 23:29:00 - [28] Originally by: Nac MacFeegle 12,000 active connections, two SQL servers... yeah, that could be an I/O problem right there... :) I'm wondering if, at this point, its more of a problem with bandwidth saturation on the SQL servers' motherboards. A PCI bus can only take so much... a northbridge can only take so much. While it may be technically correct, I just don't see two servers as being a 'cluster'. :) I'd see about form of clustering where data is distributed across multiple servers with their own disk... keep down I/O choking. This is why the system is a multi-tier one, so there aren't 12000 connections happening on the SQL servers at the same time. The beauty of having business tiers is that they buffer the database a lot. Not all the game transactions necessitate a database hit to accomplish the task. SQL Server actually isn't as bad as most people make it out to be. If you know what you're doing you can tune it extremely well, and in some cases do things more efficiently than with an Oracle database. ___winterblink/warp_drive_active/eve_nature_vraie//
Drilla	Posted - 2005.08.08 23:47:00 - [29] Solid-state drives for the SQL logs helped the E10000K Sun servers we run for db backend at Maersk. We had massive IO issues on the Oracle installations and by using solid-state disks for the SQL logs almost nullified the IO issues in one upgrade. Only downside was the pricetag on the disks but hey - Maersk isnt excatly poor when flipping out for 8 E10000K/E15000K servers at a time - all fully stocked ofc. EVE System Security - Killboard (still early alpha)
Nac MacFeegle	Posted - 2005.08.09 01:21:00 - [30] Originally by: Winterblink This is why the system is a multi-tier one, so there aren't 12000 connections happening on the SQL servers at the same time. The beauty of having business tiers is that they buffer the database a lot. Not all the game transactions necessitate a database hit to accomplish the task. Its still a hell of a lot of data to be asking two servers to read/write, even if they are beefy. Quote: SQL Server actually isn't as bad as most people make it out to be. If you know what you're doing you can tune it extremely well, and in some cases do things more efficiently than with an Oracle database. I've come to a conclusion lately: "All databases suck. Some just suck more expensively than others." Oracle's expensive and has a design and interface that still harkens to the days of mainframes, but at least it runs on various platforms. Microsoft zaps you once for the SQL Server licenses, zaps you again on the Windows Server licenses, and then if you want to have a system capable of running handling a ton of traffic you have to pony up for some beefy x86 servers... a platform that started life as a desktop computer and got shoehorned into the server niche... All opinions expressed in this message are personal and don't necessarily reflect those of the poster's corp or alliance.

Froggy	Posted - 2005.08.09 01:52:00 - [31] Originally by: Captin Biltmore I'm really supprized you answered all those questions.....I think I would be more concerned about security than appeasing the whims of a few people's interests. Especially since you have 60,000+ credit card numbers stored on your servers. Of course I am careful of not releasing any info that might cause security issues, but on the subject of security and CC info. that system is so secured that even CCP doesnÆt have access to it only the professional billing company that handles the transactions have access, doing CC transactions and securing that kind of data is what they do best so we leave that to them, we however are best at making virtual worlds work so thatÆs what we do :)
FireFoxx80	Posted - 2005.08.09 07:19:00 - [32] Originally by: Froggy Originally by: Captin Biltmore I'm really supprized you answered all those questions.....I think I would be more concerned about security than appeasing the whims of a few people's interests. Especially since you have 60,000+ credit card numbers stored on your servers. Of course I am careful of not releasing any info that might cause security issues, but on the subject of security and CC info. that system is so secured that even CCP doesnÆt have access to it only the professional billing company that handles the transactions have access, doing CC transactions and securing that kind of data is what they do best so we leave that to them, we however are best at making virtual worlds work so thatÆs what we do :) There are 60k+ accounts, but we all know GH-SC accounts for 80% of those But yeah, any company dealing with transactions normally leaves the money side of things upto someone who does it better. Loosing that many credit card numbers would happily put CCP out of business without even thinking about it. Many payment gateway suppliers are insured against that sort of thing happening. Possibly one of the 23 # ex: P-TMC \| USAC
Trevize dk	Posted - 2005.08.09 08:08:00 - [33] Originally by: SengH More importantly... how many servers power the forums? Joking aside, is CCP ever going to make the move to solid state fibre channel drives for the database? Wouldn't that solve the i/o problems overnight? Although that would be horrendously expensive..... Solid state disks. They cost a fortune. I dont see anybody using Solid State disks for the DB (even though it would be nice). Normale use would be to have the OS stored on these disks and have the different DB's on a seperat HW disk array. (unless of coz a SAN is used for the whole thing).
semp	Posted - 2005.08.09 18:25:00 - [34] "....Alteon HW loadbalancers" Alteon for the win baby. Used these when i used to work for a well known ISP in the UK (C&W) and I got to say i loved em :)
SwitchBl4d3	Posted - 2005.08.09 18:37:00 - [35] i like toast in the morning. so im problably going to need a toaster, there are two of us when i get up so im guessing a 4 slot toaster with rapid cook and defrost, no assuming theres two of us grabbing a slice im going to need butter and maybe jam. but my cunning scence tells me only one of us likes jam so im going to have to grab a piece of kitchen roll to wipe the jammy blade before doing the other slice of toast, assuming ive got a 2 slot toaster and have done the jammy toast first, thought if i plan it carefully i could do the non jammy toast then the jammy toast saving on a piece of kitchen roll.. hmmmm "Teh lord of Nonni"
SengH	Posted - 2005.08.09 18:46:00 - [36] Edited by: SengH on 09/08/2005 18:46:32 Yeah I know solid state drives are used for mission critical data that is accessed by everyone very frequently... (wouldnt suprise me if googles main site was on a solid state HD). But the I/O problems that plague eve would disappear overnight along with the hardware failures... maybe in the future I guess once CCP makes enuff money off us . Then again you could probably buy a house with each solid state HD :P Not sure if prices are still what they are but I remember someone quoting prices in the $100k range for a 40 GB solid state drive ~ 2/3 years ago.
Swebo Claw	Posted - 2005.08.09 19:00:00 - [37] I wonder if you could tell us what sort of peak bandwidth usage you see? I'm sure 12k people logged on cangenerate a lot of traffic!
Doc Brown	Posted - 2005.08.09 19:29:00 - [38] Originally by: SengH Yeah I know solid state drives are used for mission critical data that is accessed by everyone very frequently... (wouldnt suprise me if googles main site was on a solid state HD). But the I/O problems that plague eve would disappear overnight along with the hardware failures... maybe in the future I guess once CCP makes enuff money off us . Google uses their own distributed file system that they created from scratch. Do a Google search on "Google File System" to find out more than you ever wanted. _________________________________________________ There are no bad ideas, only bad implementations.
Doc Brown	Posted - 2005.08.09 19:33:00 - [39] Originally by: Swebo Claw I wonder if you could tell us what sort of peak bandwidth usage you see? I'm sure 12k people logged on cangenerate a lot of traffic! Good question.. I'm going to guess a max of about 80-90 megabytes/sec _________________________________________________ There are no bad ideas, only bad implementations.
sonofollo	Posted - 2005.08.09 20:33:00 - [40] bandwidth would be quite heavy but i think the monthly charges more than cover it

Wild Rho	Posted - 2005.08.09 20:40:00 - [41] This makes me look at my home PC and cry, and to think I thought mine kicked ass with 2MB broadband
Gavin Kineli	Posted - 2005.08.09 22:43:00 - [42] Just splurge and go for 10 SQL servers. Although I'm not quite up to par on server structure/theory and all that, striping data across more servers should increase IO output, yes?
James Lyrus	Posted - 2005.08.09 23:00:00 - [43] Originally by: Trevize dk Originally by: SengH More importantly... how many servers power the forums? Joking aside, is CCP ever going to make the move to solid state fibre channel drives for the database? Wouldn't that solve the i/o problems overnight? Although that would be horrendously expensive..... Solid state disks. They cost a fortune. I dont see anybody using Solid State disks for the DB (even though it would be nice). Normale use would be to have the OS stored on these disks and have the different DB's on a seperat HW disk array. (unless of coz a SAN is used for the whole thing). Solid state disk tech is somewhat overkill. If you look at the high end SAN storage arrays, e.g. the Symmetrix DMX you'll see something that's deeply sick for doing disk IO. Fully populate your PCI bus with dual channel host bus adaptors and you can be doing disk IOs at 8Gb/sec/card, and have enough io cache on the back end to support doing that indefinitely. (A 'large' Symmetrix has cache measured in gigs, so it's more than possible to have enough cache for the entire volume to effectively be running off memory.) Course, the price tag starts to rack up when you start looking in that direction, but I wouldn't be too suprised to find that those IO intensive servers were hooked up to some midrange SANs. Fiber channel disks, supported by very large read and write caching are pretty hard to beat.
Coltar	Posted - 2005.08.19 23:28:00 - [44] And just to give you a warm fuzzy feeling last time I checked Microsoft charge approx ú3000 per processor for a SQL enterprise edition licence so one server costs over ú12000 in licences.
Mr Popov	Posted - 2005.08.20 05:08:00 - [45] Originally by: Coltar And just to give you a warm fuzzy feeling last time I checked Microsoft charge approx ú3000 per processor for a SQL enterprise edition licence so one server costs over ú12000 in licences. Let's pool our isk together, and mabye GH-SC can be persuaded grief Bill Gates into giving CCP the license for free.
Serilla	Posted - 2005.08.20 07:48:00 - [46] very neat to hear about this stuff. Hopefully we can get some more hardware upgrades that way the coders can focus more on content than making the game work with the limited hardware. like to see more about how the IO issues will be addressed to make all our data requests go through in less ticks


Pages: 1 2 :: [one page]

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.