Pages: 1 [2] 3 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 11 post(s) |
Darth Sith
SiN. Corp Sons of Tangra
|
Posted - 2009.06.11 05:10:00 -
[31]
Originally by: Morphisat Do you run the servers in core mode ? Would reduce some overhead and it's a nice new feature of server 2008.
Last time I checked, SQL 2005 or 2008 were not supported by MS in core mode....
|
Darth Sith
SiN. Corp Sons of Tangra
|
Posted - 2009.06.11 05:17:00 -
[32]
Originally by: Entipathy @The Lag People: It will likely not help much as the fights don't directly hit the database. However the news that they finished upgrading the SOL nodes will help the lag. The biggest limiter in the fleet fights now is both client side rendering and sheer brute calculations server side. The next big step they need to take is to make nodes able to span multiple processor cores.
This might be accomplished using VMware ESX, not sure if CCP has looked into this technology but it might resolve some of your single theard issues.
ESX just allows you to put a whole lot of O/S instances onto the hardware. It really has no benifit to a single threaded application other then to allow you to run many of the workloads in isolation on the same chunk of hardware. On earlier blogs CCP has already alluded to the fact that they are running many solar systems on a single node indicating that they are using processor affinity to run multiple single threaded instances on one O/S image without a hypervisor. While I am a huge fan of ESX it would only introduce more issues then it is worth as only CPU workloads execute with very little overhead. Both disk and network transactions inherit a level of increased latency due to having to mess with virtualized adapters.
|
El Covah
LOCKDOWN. Cult of War
|
Posted - 2009.06.11 07:09:00 -
[33]
I am curious, what are you doing with all the (still usable) good old hardware? Is it beeing used for test server? If you are open for donations I am ready to receive some
El.
|
Entipathy
|
Posted - 2009.06.11 07:10:00 -
[34]
Edited by: Entipathy on 11/06/2009 07:12:21 network adapters on ESX operate at the same rate as the CPU and if you are like my company and invest into fiber disk arrays you bandwidth is almost 2 x that of any current server based technology. As far as single theard application ESX allows you to utilize all cpu as if they are one. So in the case of my setup below we have 21000Mhz. Avaliable which can be shared or used by one virtual machine.
I personally manage and ESX environment with 6 IBM 3650 with over 6 terabyes of 4gb fiber storage. I'd like to think I know what I am talking about.
This is what one of our servers looks like http://img269.imageshack.us/img269/8706/esxftw.jpg |
Maarek Aramin
|
Posted - 2009.06.11 07:18:00 -
[35]
Originally by: El Covah I am curious, what are you doing with all the (still usable) good old hardware? Is it beeing used for test server? If you are open for donations I am ready to receive some
El.
I'd hazard a guess they'll use it for the new MMO they've been developing after merging with White Wolf back in 1996. Pretty sure it'll be based on the World of Darkness IP which includes Vampire: The Requiem.
- Maarek |
Rivqua
Caldari Omega Wing R.E.P.O.
|
Posted - 2009.06.11 07:24:00 -
[36]
Quote: ...but we had to run 32-bit processes on machines that had 4 GB RAM due to the RAM usage overhead of 64-bit processes.
Does that mean you still have some 4gb ram /w 32Bit sol blades, or was that just not optimal phrasing, saying that before the change, you had to run 32BIT on the 4GB Opteron nodes?
- Riv _________________ - Rivqua - --- R.E.P.O. --- |
Kyoko Sakoda
Caldari Veto. Veto Corp
|
Posted - 2009.06.11 07:47:00 -
[37]
Very impressed by the blog, but I had to go out and buy my own cookie.
___
Latest video: War Has Come (720p) |
Yaar Podshipnik
Gallente Paxton Industries Paxton Federation
|
Posted - 2009.06.11 08:11:00 -
[38]
I love reading about how you do things on TQ. Thanks for sharing!
|
Nova Fox
Gallente Novafox Shipyards
|
Posted - 2009.06.11 09:27:00 -
[39]
Would not surprise me if thier older gear winds up in test server and possibly a 'terra' cluster for WIS.
Pre-order your Sisters of ≡v≡ Exploration ship today, Updated 25MAY09 |
Fumen
|
Posted - 2009.06.11 09:28:00 -
[40]
"Must. Have. Cookies, Scottie!"
"I'm given 'er all she's got, cap'n. Oven won't bake any faster!"
|
|
|
CCP Valar
|
Posted - 2009.06.11 10:31:00 -
[41]
Originally by: Chribba Edited by: Chribba on 10/06/2009 19:59:59
Valar new? Ok...? hehe
Anyway I love the hardware, I love the databases. I love CCP. Can I have your old RamSan?
and oohhh ohhh /edit/smaller downtime rocks!! Keep up the great work!
I'm sorry Chribba, we are still using the old ones for our hottest objects, as they perform better than the new one. If we weren't still using them, I'd of course seriously consider giving them to you.
Originally by: Entipathy There was no comments if these issues will resolve the lag issue we get during large engagements, will these hardware upgrades do anything to resolve the lag issue?
Upgrades to the database hardware only have a limited effect on fleet fight lag, as those are mostly bound by sol server CPU. The Wolfdale upgrades should help with fleet fight lag, but fleet fights that we have recieved timely notifications about have been running on Wolfdales for some time already, so there might not be a noticable difference.
Originally by: IceBlade Rage @Fallout: I hope you mean new relatively, as I think Valar has been around longer than you have, at least publicly.
@Valar: Pictures Please!
@The Lag People: It will likely not help much as the fights don't directly hit the database. However the news that they finished upgrading the SOL nodes will help the lag. The biggest limiter in the fleet fights now is both client side rendering and sheer brute calculations server side. The next big step they need to take is to make nodes able to span multiple processor cores.
Overall, awesome blog. We need more like this heh.
Heh, Fallouts comment sounded a bit strange, I've been with CCP for almost 5 years now, but I was a devblog virgin. About the picture request. We can't show you pictures of Tranquility, as cameras are not allowed on the data floors of the hosting center.
Originally by: Morphisat Do you run the servers in core mode ? Would reduce some overhead and it's a nice new feature of server 2008.
We don't run the database servers in core mode, but most of the sol servers are running core.
Originally by: Mashie Saldana Speaking of "new" database admins, is CCP Sharkbait still around?
How many database request do you get per day now, I know a year or so ago it was around 550 million a day.
CCP Sharkbait is still around but has never been a database admin, nor a part of the virtual world administration team. He's currently an Assistant Technical Producer in the Core programming department. Before that he was a part of the QA department. He has however helped with running TQ and been very visible on the forums about TQ issues in the past.
The database request number is actually a little bit lower according to the performance counter I'm looking at right now. I might not be looking at the same metric as you are referring to though, but it might also be because of optimization done by our programming department. |
|
|
CCP Valar
|
Posted - 2009.06.11 10:34:00 -
[42]
Originally by: Ford Chicago Lots of great info, but I'm still curious!
Why are you only using two CPUs per cluster node as opposed to four? Upgrading to four seems like a relatively low cost even considering additional SQL Server licensing costs.
Are you using 32x4GB RDIMMS or 8GB RDIMMS?
What is the average processor load during downtime/startup? What is the average processor load during peak user activity?
I know you're using multi port FiberChannel HBAs to the RAMSAN, but what connects the database server to the sol nodes? Are those connections anywhere near saturation?
Again, great tech blog.
Having 2x 6 core CPUs is a nice, cost efficent upgrade, that doesn't require us to upgrade our SQL Server license. While I understand why you are asking, the benefits of more CPUs are obvious, we are running at less than 20% CPU at the moment. We spend a little bit less money on idle CPUs and can spend more on sol servers or other YARRDWARE! Well, you see what I'm getting at. We have 32x4GB RDIMMs and have gigabit ethernet connecting the database to the sol servers. That connection is under 20% utilization. We like to have large margins everywhere.
Originally by: MotherMoon wait, ow, confused.
ok so before you had 4 3.3 GHz Wolfdale CPUs and 16 GB of RAM blades running right?
and like for a fleet flight you had to ask before hand to have a system placed on one to fight the lag.
Am I to believe that you now have nothing but 3.3 GHz Wolfdale CPUs and 16 GB of RAM nodes? Or am I understanding wrong?
also does this mean we no longer have to ask to reinforce a system? because every system in eve is this beefy??
You are right, sometime last year, we had only 4 Wolfdales, that we used for fleet fights. In december we upgraded half the cluster and few weeks ago we upgraded the rest. Now all of the cluster is running on Wolfdales. You still have to send in fleet fight notifications so we can dedicate a node for a fleet fight, as otherwise the solarsystem might be on a node with a whole constellation mapped on it as well.
Originally by: Maria Kalista I hope one day you can show us how the EVE cluster looks like, with some pretty database babe in front of it.
So, when will this upgrade happen?
The only part we have left is to start the replication and do an in-place upgrade to SQL Server 2008. This is in progress. We've had some difficulties in starting the replication and that actually caused the node deaths, traffic advisories and market issues on tuesday afternoon. We are working with Microsoft on the issue.
Originally by: El Covah I am curious, what are you doing with all the (still usable) good old hardware? Is it beeing used for test server? If you are open for donations I am ready to receive some
El.
Our old hardware is used for test servers, webservers and internal office servers. Don't worry, we can put it to good use.
Originally by: Rivqua
Quote: ...but we had to run 32-bit processes on machines that had 4 GB RAM due to the RAM usage overhead of 64-bit processes.
Does that mean you still have some 4gb ram /w 32Bit sol blades, or was that just not optimal phrasing, saying that before the change, you had to run 32BIT on the 4GB Opteron nodes?
- Riv
Before we replaced the second half of the cluster we had to run 32 bit processes on the Opterons, because the Opterons have only 4GB of RAM. 64-bit processes have a lot of RAM overhead. There are no Opterons in the cluster anymore.
|
|
Altaree
Eve University Ivy League
|
Posted - 2009.06.11 10:54:00 -
[43]
YAY GEEK ****!! WOO WOO!!! I am going to wander off to my hanger and get that cookie now.. --Altaree Eve University |
Batolemaeus
Caldari Free-Space-Ranger Morsus Mihi
|
Posted - 2009.06.11 11:03:00 -
[44]
Originally by: CCP Valar About the picture request. We can't show you pictures of Tranquility, as cameras are not allowed on the data floors of the hosting center.
Hand painted MsPaint scetch? ----------------------------------------------
Originally by: CCP Prism X In New Eden, EVE wins you.
|
Eleana Tomelac
Gallente Eclats de verre
|
Posted - 2009.06.11 12:29:00 -
[45]
Originally by: Batolemaeus
Originally by: CCP Valar About the picture request. We can't show you pictures of Tranquility, as cameras are not allowed on the data floors of the hosting center.
Hand painted MsPaint scetch?
This is just the background, we still want all of CCP's girls on the picture! |
Louis deGuerre
Gallente Azure Horizon Federate Militia
|
Posted - 2009.06.11 12:38:00 -
[46]
Nomnomnom...cookies !
This techtalk from Valar has made me all warm and fuzzy inside. The content is all good. I for one welcome our new DB overlord Valar ! Don't forget to give us a heads up for that SQL server upgrade so I can set a 30-day skill --- Sol: A microwarp drive? In a battleship? Are you insane? They arenĘt built for this! Clear Skies - The Movie |
Che Biko
Polytechnique Gallenteenne
|
Posted - 2009.06.11 13:16:00 -
[47]
Where are the cookies?
|
Robot Robot
|
Posted - 2009.06.11 13:25:00 -
[48]
This is how you do things.
If you ever need an assistant DB administrator, I would consider getting back into the industry AND moving back to Europe to work under this man.
|
gtiness
|
Posted - 2009.06.11 13:48:00 -
[49]
All sounds like fun (er, kinda). have to love SQL tho. Great tool. Are you working with the SQL Team at all? I'm guessing they'd love to work with you guys.
Can't help but notice those fancy new servers officially support Windows Server 2003, and y'all got 2008 on'em already
|
TornSoul
BIG Libertas Fidelitas
|
Posted - 2009.06.11 14:06:00 -
[50]
Originally by: CCP Fallout
Originally by: TornSoul And "new" and "Valar" in the same sentence (even though Fallout made a nice "save" with the "blogging community) is just lol
No cookie for you :D
TouchT
|
|
Toksyuryel
Gallente Domestic Tentacle Supply
|
Posted - 2009.06.11 14:23:00 -
[51]
Windows...? Microsoft...? For a server? Now I see where a lot of these problems originate. Y'all needs to get you some BSD and PostgreSQL on those boxes ASAP :P |
Dead6re
Amarr Ministry of War
|
Posted - 2009.06.11 14:41:00 -
[52]
Originally by: CCP Valar
You are right, sometime last year, we had only 4 Wolfdales, that we used for fleet fights. In december we upgraded half the cluster and few weeks ago we upgraded the rest. Now all of the cluster is running on Wolfdales. You still have to send in fleet fight notifications so we can dedicate a node for a fleet fight, as otherwise the solarsystem might be on a node with a whole constellation mapped on it as well.
Will EVE ever be able to relocate resources across the nodes, so it would detect a fleet battle and move the other systems in the constellation to other nodes?
Quote: CCP Sharkbait: i'm using vista 3bit business and i have no problems at all with most things really, including Eve.
|
Andrea Griffin
|
Posted - 2009.06.11 19:10:00 -
[53]
I wish the company I worked for had people on top of things like this. Our databases are not happy. They are neglected and abused. It terrifies me when I hear things like, "Oh, replication broke again?" It isn't like we serve up 10-15 million page hits a day, but... ..oh wait, we do? Crap.
Having tech-centric blogs is fun and awesome, and I would love to have more. I don't really care what they are about either. They're always full of good information and it's great to get an inside look at things.
Too bad about not having cameras in the NOC, though. Personal request though. If people are asking for pictures of the hot CCP chicks, then I want pictures of the hot CCP dudes, too. |
Darth Sith
SiN. Corp Sons of Tangra
|
Posted - 2009.06.12 00:01:00 -
[54]
Originally by: Entipathy Edited by: Entipathy on 11/06/2009 07:24:12 Network adapters on ESX operate at the same rate as the CPU and if you are like my company and invest into fiber disk arrays you bandwidth is almost 2 x that of any current server based technology. As far as single theard application ESX allows you to utilize all cpu as if they are one. So in the case of my setup below we have 21000Mhz avaliable which can be shared by many or used by one virtual machine.
Other features that ESX offers are the ablility to move virtual machines between hosts on the fly as well as moving them between datastore these technology are called Vmotion and Storage Vmotion. Which would be good for reinforcing nodes as you could move any other server off the hardware.
I personally manage and ESX environment with 6 IBM 3650 with over 6 terabyes of 4gb fiber storage. I'd like to think I know what I am talking about.
This is what one of our servers looks like http://img269.imageshack.us/img269/8706/esxftw.jpg
I dont presume to understand the complexity of tranquility, but CCP if you haven't already looked into ESX then I highly suggest you do.
Um .. well if we're whipping our epeens out, I am the senior most scalable systems / virtualization specialist for IBM in Canada with one customer alone running 2700 vm's on 3850 m2's and collectivly have architected systems for over 10,000 vms. That said:
"Network adapters on ESX operate at the same rate as the CPU"
Only in new ESX 4 with VT for I/O where you can allocate a Nic physically to a VM. Downside is that the moment you do that you can no longer do the fancy stuff like VMotion
"ESX allows you to utilize all cpu as if they are one"
UM .. That one made me LOL .. no each vm executes on discreet cores. Only way for a vm to execute on multiple cores is to allocate multiple vCPU's to the VM which will, if not done correctly on a system with enough cores cause scheduler latency issues as the vm can only execute when enough cores are avalable to service the number of vpcu's alocated to the vm. No you do not get a dozen cores showing up as a phat huge proc just because the performance counter measures things in available MHZ :)
"if you are like my company and invest into fiber disk arrays you bandwidth is almost 2 x that of any current server based technology."
Actually fibrechannel is SCSI 3 over a fibre protocol. so you actually inherit a slight latency encapsulating over the SAN to talk to the disks. So disk for disk, direct attach is actually faster however SAN connectivity gives you all the sweet capabilities like mapping the vmfs luns to multiple hosts for parallel filesystem access.
Anyone interested in more detail can always fire me a mail in game. Out of respect for this thread and because I have already hijacked enough of this thread this will be my last post here.
Darth
|
Danag Darklightning
Caldari
|
Posted - 2009.06.12 09:53:00 -
[55]
That was a most excellent read. As an SA at a large University I'm all about bigger and faster when it comes to our data-center, and CCP; you most undoubtedly know all about bigger and faster, LOL.
|
Zerakix
Minmatar LEAP Corp
|
Posted - 2009.06.12 13:43:00 -
[56]
CCP Valar,
I'm guessing the only reason you didn't opt to use a Nehalem-EP based IBM sever for the DB is the 64GB of ram they max out at. I do wonder why you didn't op to use them for the other half of the Sol servers given the likely boost they would offered over the wolfdale DP based hardware. I fail. |
|
CCP Mindstar
|
Posted - 2009.06.12 16:47:00 -
[57]
Originally by: Zerakix CCP Valar,
I'm guessing the only reason you didn't opt to use a Nehalem-EP based IBM sever for the DB is the 64GB of ram they max out at. I do wonder why you didn't op to use them for the other half of the Sol servers given the likely boost they would offered over the wolfdale DP based hardware.
I might just jump in and take a minor portion of the Valar thunder here and answer this question on our blade servers...
We have been testing the Nehalem hardware in our cluster and it is very promising, but the HS22 blades were not even available at the time we decided to get latest group of wolfdales (I'm pretty sure it was the case when we ordered our new DB servers too).
We will no doubt be keeping a close eye on the performance we get from our test hardware - and if there is a case to be made for an upgrade I am sure it will be done
Certainly, there are some interesting options on the horizon and the "Turbo mode" feature is really cool. |
|
Elaron
Jericho Fraction The Star Fraction
|
Posted - 2009.06.12 20:45:00 -
[58]
With Nehalem-EP breaking TPC-C records and the coming Nehalem-EX being an obscenely powerful lump of silicon, it's a good time to be evaluating this kind of hardware.
|
Eventy One
Magellan Exploration and Survey
|
Posted - 2009.06.13 00:47:00 -
[59]
An answer here, might be a nice cookie. |
OneT
Wicked Weasels
|
Posted - 2009.06.13 00:55:00 -
[60]
Originally by: CCP Valar
Originally by: Chribba Edited by: Chribba on 10/06/2009 19:59:59
Valar new? Ok...? hehe
Anyway I love the hardware, I love the databases. I love CCP. Can I have your old RamSan?
and oohhh ohhh /edit/smaller downtime rocks!! Keep up the great work!
I'm sorry Chribba, we are still using the old ones for our hottest objects, as they perform better than the new one. If we weren't still using them, I'd of course seriously consider giving them to you.
I personally vote that all hardware chibbra could use that ccp is currently using to keep the beer in there fridge at a perfect temperature (or other important tasks like running the office counterstrike server / media server) should be handed over to him right away
|
|
|
|
|
Pages: 1 [2] 3 :: one page |
First page | Previous page | Next page | Last page |