| Author |
Thread Statistics | Show CCP posts - 5 post(s) |
|
Oveur
   |
Posted - 2006.05.26 20:24:00 -
[1]
 Originally by: Jake Udunno
argh!, just when I actually get into the game they reboot servers.
Its a conspiracy I tell ya!!! Not sure which conspiracy, give me a while to think one up.
This is not the conspiracy you are looking for.
In other news.
Don't use your superpowers to destroy solar systems.
Senior Producer
EVE Online
|
|
|
Oveur
|
Posted - 2006.05.26 20:41:00 -
[2]
Originally by: branodn lee
ok lets just say whats really happening here. ccp is spending so much time with the new china server that we are getting the very short end of the stick. this is the 2nd node death in 2 weeks and if im right the one last week was on the same day as this one. yet ccp has said nothen about what the deal is or if they are looking into it. ever since they started working on the other server we here on TQ are getting left in the dark and not told anything about the problums our server is haveing. so please ccp, devs, gms or anyone that can give a reason for why this node death is happening everyone week now would be nice.
Most of it is actually due to our extensive hardware upgrades done to Tranquility in the last months. Practically everything is new or has been upgraded in the last three months - covering both hardware and software.
New stuff development for Tranquility suffering because of China? Yes, I pointed that out some time ago.
Tranquility stability and fixing suffering from China? No. It's growing pains. EVE doubled in the last year.
What drew us down today, that's still being investigated, first order of business was to get the business back in order 
Senior Producer
EVE Online
|
|
|
Valar
|
Posted - 2006.05.26 21:09:00 -
[3]
A little post-mortem.
At 20:01 we lost all nodes except 13 and all players were dropped from the cluster. As players started to log in, resources got remapped and reloaded.
As soon as I was able to get to my computer, and connect to the cluster to find out what was wrong, I initiated an emergency reboot, but at that time we had over 13k players running on only 13 nodes 
Thats incredible that the 13 nodes survived the load of 13.000 players logging in.
As soon as the cluster was down and cleanup tasks were done I started the cluster again.
The server was up and accepting connections less then 20 minutes after the original node crash and less then 10 minutes after I arrived.
This incredible rebooting time is due to the new SQL hardware and a hotfix that fixes distributed compression of serverlogs. After getting two fixes I have been waiting for, one with the patch and one as a hotfix yesterday I was able to change the accept delay of the cluster to 2 minutes instead of the 10 minute wait it was before and increase the throttling speed by 300 connection per minute.
The fix for distributed compression also makes cleanups after shutdowns much quicker.
Well, since I started the cluster back up I've been looking into what happened.
While I can't be exacly sure what happened at this point the serverlogs show what looks like a mass disconnection between nodes internally... however the logs also say that the connections being dropped weren't actually supposed to be there...
I guess I'll spend a big part of the evening playing detective.
------
Valar
Database admin - Server operations team
CCP Games
How to write a good bugreport |
|
|
Valar
|
Posted - 2006.05.26 22:37:00 -
[4]
An update. I found out what caused this.
After startup today, a proxy process went into a state where it kept erroring.
At 20:01 it started dropping its connections and each connection to a sol server that dropped caused an event that broadcast a node death for the node that was connected to the disconnected transport.
So the node death detector on a proxy process that was broken caused the server to kill almost all its nodes.
I'll forward this info to PapaSmurf and porkbelly and they will hopefully find a way to prevent this from happening in the future. Thankfully, proxy deaths are very rare.
------
Valar
Database admin - Server operations team
CCP Games
How to write a good bugreport |
|
|
Valar
|
Posted - 2006.05.27 00:57:00 -
[5]
Originally by: Michiyo Daishi
Originally by: Valar
Thankfully, proxy deaths are very rare.
umm rare eh? XD someone get the probability mathematicians in here!
Seriously though, great work :D CCP at it again!
Now where's my cookie?! XD
Proxy deaths are very rare and in 50% of cases are very visible to the players due to limitations of our load balancer. Sol node deaths are however more common.
------
Valar
Database admin - Server operations team
CCP Games
How to write a good bugreport |
|
| |
|