Originally by: stoats

---

"We don't know, but we're working on a fix"

---

Originally by: Kuseka Adama

---

This wasnt just servers. This was the website/forums I think something a lot bigger than a simple glitch happened tonight guys...

---

Originally by: Ataractic

---

Hacker? A definate possibility.....

---

Originally by: Tobias Sjodin

---

Oveur got drunk and tripped on a cable again.

---

Originally by: SasRipper

---

"the usal party @ ccp hq went slighly wrong 2 meambers who shall not be named set about pleasuring each other in the sever room the resulting bodyly fluids caused a some fusing to the sever which resulted in small fire the sever has died eve is over "

---

Originally by: CCP Sharkbait

---

we are screwed. delaying startup again.

---

Originally by: Tobias Sjodin

---

Oveur got drunk and tripped on a cable again.

---

Originally by: SasRipper

---

"the usal party @ ccp hq went slighly wrong 2 meambers who shall not be named set about pleasuring each other in the sever room the resulting bodyly fluids caused a some fusing to the sever which resulted in small fire the sever has died eve is over "

---

Originally by: CCP Sharkbait

---

we are screwed. delaying startup again.

---

Originally by: KaiserSose1

---

I swear to god,
If i have to talk to my wife on a friday night because EVE is offline...
SOMEBODY is gonna get GANKED!

---

Originally by: KaiserSose1

---

I swear to god,
If i have to talk to my wife on a friday night because EVE is offline...
SOMEBODY is gonna get GANKED!

---

Originally by: AirWalker

---

i keep saying ditch the gerbils and hamsters....use wolverines in heat...they never quit

---

Originally by: AirWalker

---

i keep saying ditch the gerbils and hamsters....use wolverines in heat...they never quit

---

Originally by: Cantrev

---

Starting up, 501 seconds....HA!

---

Originally by: KaiserSose1

---

I swear to god,
If i have to talk to my wife on a friday night because EVE is offline...
SOMEBODY is gonna get GANKED!

---

Originally by: Paddlefoot Aeon

---

Gentlemen, tonight is a sad night.
Tonight, we must say goodbye to speedy... the EVE-Online cluster hamster.

However, he has not passed on... he was sent to jail for violating his probation. Aparently he was operating a server with a suspended license. He will be serving anywhere from 23 to 45 days in a minimum security prison.

He wanted a solo cell... but he ended up sharing a cell with Paris Hilton.

..and let me tell you, you should have seen the dirty look that Paris gave Speedy. One wonders what Paris will do, alone with a Hamster, for 45 days.

Gentlmen (and ladies), please join me in wishing Speedy the very best of luck.

---

Originally by: Sarper Andrenti

---

328 secs

---

Originally by: Besmin

---

I have 147 macro miners and ISK farmers in my buddy, list, when i log on in a sec i wonder how many will show up on the right?

---

Originally by: AirWalker

---

Edited by: AirWalker on 09/06/2007 02:18:50

Originally by: Besmin

---

I have 147 macro miners and ISK farmers in my buddy, list, when i log on in a sec i wonder how many will show up on the right?

---

you won't have a screen to play in

---

Originally by: demonfurbie

---

yarrr .... to the test server ... unless they are starting up in 600 sec every 600 sec

---

Originally by: Bryce Bolz

---

This happens every freaking friday I am sick of it we already pay for 30 hours we cant use each month now this!! I lost my Iteron Mk V due to lag it had 5 t2 expanders. I am also willing to bet that the gm's will lock this thread like last time because they don't want to listen to their paying customers complain because they are not doing their job.

---

Originally by: Arcadia1701

---

wow goes down for * emergecny maintinence* and eve follows within 5 mins with a server crash w00t.

---

Originally by: Reverof

---

This is stupid why have the startup timer resetting at 0?

Hope you devs realize you just need to set a realistic time, if its 30 mins then fine don't loop numbers at us !

Get it back up your costing me money

---

Originally by: DvlDwg

---

I would think that a gaming entity so large would have at the very least a basic fundamental understanding of good customer service. How about a news flash or something? Some sort of announcement.. appology.. something??

I left the last game I played for the same type of crap.

---

Originally by: Shameless Avenger

---

- Me : Honey, let's watch a movie together!
- Wife: Hum, is your game is down again?
- Me : Erhm... kind of.

---

Originally by: AirWalker

---

Originally by: DvlDwg

---

I would think that a gaming entity so large would have at the very least a basic fundamental understanding of good customer service. How about a news flash or something? Some sort of announcement.. appology.. something??

I left the last game I played for the same type of crap.

---

fix > apology and fake sincerity

---

Originally by: De Cairo

---

EvE is a badly made game patched together with band-aids. The on-going design changes are reactive (easy) rather than pro-active (professional). It can only get worse unless there is a complete re-write. It just so happens that it's very fun to play.

---

Originally by: CCP Wrangler

---

You're not supposed to feel like you're logging in to a happy, happy, fluffy, fluffy lala land filled with fun and adventures, thats what hello kitty online is for.

---

Originally by: CCP Valar

---

Postmortem for tonights issues.

Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration.

This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues.
This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs.

The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure.
The server was started back up but it quickly became apparent that this was still broken under load.

At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again.
When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan.
However when the server was back up the same thing happened again.

I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures.

When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server.
Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load.

The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly.

Signing off(and going back to bed)

---

Originally by: CCP Valar

---

Postmortem for tonights issues.

Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration.

This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues.
This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs.

The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure.
The server was started back up but it quickly became apparent that this was still broken under load.

At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again.
When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan.
However when the server was back up the same thing happened again.

I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures.

When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server.
Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load.

The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly.

Signing off(and going back to bed)

---

Originally by: Amarria Black

---

Ductoris, you keep pointing people here, and this last post from Valar (over an hour ago) is phrased in such a way as to imply the problem is fixed. It's not. The server may be up and running, but it's shakier than a 104 year old man with Parkinson's. Dock / undock times in the tens of seconds, autowarping to some random point 1mil KM from gate on system jump-in, jump-out lag in the tens of seconds, jump queues at 2-3 minutes for empty systems.

So, can we get a REAL update?

---

Originally by: Siri Blue

---

I come back from work...and am drunk...and WTF EVE IS DOWN!?!?!??!?!

And I bet its some "oh, well, some admin forgot to clear some cash issue" so we d'oh...have to clear some and reboot...and crap""""

Get some people to do the daily maintenance who actually do the maintanance and don't just play minesweeper on the servers while they reboot!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! (111)

---

Originally by: Ramius Decimus

---

Edited by: Ramius Decimus on 09/06/2007 07:58:53

Originally by: Amarria Black

---

Ductoris, you keep pointing people here, and this last post from Valar (over an hour ago) is phrased in such a way as to imply the problem is fixed. It's not. The server may be up and running, but it's shakier than a 104 year old man with Parkinson's. Dock / undock times in the tens of seconds, autowarping to some random point 1mil KM from gate on system jump-in, jump-out lag in the tens of seconds, jump queues at 2-3 minutes for empty systems.

So, can we get a REAL update?

---

That's strange, because I wasn't having any of the problems you were. And I was even in Jita. Must be all on your end.... or the node your system is on...

Woot, server back up!!! Cheers, CCP....

---

Originally by: CCP Valar

---

I called time of death for TQ as soon as I saw its status and a shutdown was initiated again.

...


When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server.
Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load.

---

Originally by: Rauth Kivaro

---

Temporary glue applied. Two in one day for maruud.

---

Originally by: parrowdox

---

i was autopiloting, i got to the jita gate then everything froze for 5 mintues i try to log an it wydnt let me go to my character :(

---

Originally by: DvlDwg

---

I'm not upset that the servers were down. I understand that servers going bad, data corrupting, somebody restarting something when they shouldn't is going to happen. What gripes me is the obvious lack of concern about NOTIFYING the player database and the, in my opinion, complete lack of unproffessionalism in locking the forums so no one can even read a message or post to get a time frame or hint of when things will be corrected.

Bad form... very bad form. Post something from the get go. Notify your paying customers there is a problem, you expect it to be fixed by such and such time, and then post what the problem is after it's fixed.

I'm not so addicted to this game that I won't stop giving you my $15 a month. If you don't care about me... I can assure you that I will quit caring about you very quickly.

---

Quote:

---

Experiancing Database Issues,

---

Originally by: Marduk Felzhen

---

Originally by: Vorian Atraties

---

i bid my kidney...

---

i bid his other kidney

---

Originally by: DvlDwg

---

I'm not upset that the servers were down. I understand that servers going bad, data corrupting, somebody restarting something when they shouldn't is going to happen. What gripes me is the obvious lack of concern about NOTIFYING the player database and the, in my opinion, complete lack of unproffessionalism in locking the forums so no one can even read a message or post to get a time frame or hint of when things will be corrected.

Bad form... very bad form. Post something from the get go. Notify your paying customers there is a problem, you expect it to be fixed by such and such time, and then post what the problem is after it's fixed.

I'm not so addicted to this game that I won't stop giving you my $15 a month. If you don't care about me... I can assure you that I will quit caring about you very quickly.

---

Originally by: brinelan

---

Victory not Vengeance [VnV]

---

Originally by: Kuseka Adama

---

This wasnt just servers. This was the website/forums I think something a lot bigger than a simple glitch happened tonight guys...

---

Originally by: CCP Valar

---

Postmortem for tonights issues.

Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration.

This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues.
This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs.

The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure.
The server was started back up but it quickly became apparent that this was still broken under load.

At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again.
When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan.
However when the server was back up the same thing happened again.

I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures.

When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server.
Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load.

The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly.

Signing off(and going back to bed)

---

Originally by: Haffrage

---

One does not simply log in to Jita.

---