| Author |
Thread Statistics | Show CCP posts - 9 post(s) |
|
GM Faolchu
Game Masters
   |
Posted - 2007.05.25 01:54:00 -
[1]
The server is currently experiencing database problems, the server team is looking into it. The server should be up and running shortly without any further issues.
We apologise greatly for any inconvenience caused and hope that there will be no further issues.
|
|
|
CCP Admiral Chamrajnagar
|
Posted - 2007.05.25 02:09:00 -
[2]
Server is now online, we had a database problem where the database server went too 100% cpu and remained there for 1 hour. I made the decision after consulting with the GM's to take the server down gracefully instead of allowing it to melt down.
A job had caused an abnormal load and the server was trying to catch up but could not.
Upon startup of the cluster again a machine decided to reboot itsself and cause the infinite startup loop.
After correcting that issue we are now online.
My apologies for the delay in updating you.
|
|
|
Ductoris
Forum Moderator
Interstellar Services Department
|
Posted - 2007.05.25 02:17:00 -
[3]
 Originally by: Dapanman1
Wow....way to ruin our fun...you guys locked our thread =/
GM threads usually take priority when deciding which thread will be the official thread. Had Faolchu not posted I would taking the first thread, which was admittedly yours.
Now it's time to clean up GD so I'm locking the other threads, nothing personal.
|
|
|
Ductoris
Forum Moderator
Interstellar Services Department
|
Posted - 2007.05.25 02:22:00 -
[4]
Originally by: Dapanman1
/me thumbs nose at GMs
I don't like them much. They do nasty things to me on IRC.
I can't tell you but I can show you on the doll......
|
|
|
Ductoris
Forum Moderator
Interstellar Services Department
|
Posted - 2007.05.25 02:25:00 -
[5]
Originally by: Dapanman1
pst...thumb your nose at them
Done.
He didn't like it one bit.
|
|
|
Kreul Intentions
Forum Moderator
Interstellar Services Department
|
Posted - 2007.05.25 04:13:00 -
[6]
/me smacks the hamsters around a bit.
Re-stuck!
|
|
|
Huitzilopochtli Tlaloc
Forum Moderator
Interstellar Services Department
|
Posted - 2007.05.25 08:26:00 -
[7]
Server is back up and running. Im sure there will be a official announcement about the problems at some point today.
- Thanks Hutch.
____
forum rules | [email protected] | ME
They call me Hutch. I have forgotten why  |
|
|
CCP Valar
|
Posted - 2007.05.25 08:47:00 -
[8]
A quick post mortem after this horrible night.
The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack.
This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem.
After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables.
The SQL Server is now back to its normal self. The development team will hopefully have a proper fix for this ready soon.
And now back to bed 
------
Valar
Database admin - Server operations team
CCP Games
How to write a good bugreport |
|
|
Yipsilanti
Forum Moderator
Interstellar Services Department
|
Posted - 2007.07.12 02:10:00 -
[9]
This conversation has been over for quite a while, please do not resurrect old threads
___
|
|
| |
|