EVE Search - Server issues

All Channels

EVE General Discussion

Server issues

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 3 4 [5] :: one page
Author	Thread Statistics \| Show CCP posts - 3 post(s)
Shameless Avenger	Posted - 2007.06.09 06:51:00 - [121] Valar, we love ya!
Drizit Amarr	Posted - 2007.06.09 06:55:00 - [122] Edited by: Drizit on 09/06/2007 06:56:33 Well done for figuring it out so fast Valar. At least you were straight with us and told us what was happening. ------------------------------------ Originally by: De Cairo EvE is a badly made game patched together with band-aids. The on-going design changes are reactive (easy) rather than pro-active (professional). It can only get worse unless there is a complete re-write. It just so happens that it's very fun to play. @ De Cairo: snip Not necessary -Rauth This sort of thing does not happen often and considering that we have over 30 thousand users on one server at peak times, I think that is something of a success for Eve. Try playing some of the other MMO's sometime, they can fall over with major lag problems with just 5 thousand or less on the server. WOW goes down so frequently that you can almost set your watch by it. --
Jim McGregor	Posted - 2007.06.09 06:56:00 - [123] Edited by: Jim McGregor on 09/06/2007 06:55:54 I sure wouldnt want to work as database admin for this game... getting a call in the middle of the night, going to work to fix it, spending hours trying to find whats wrong when having just a few hours of sleep under your belt... and then also posting in the forums to inform the community what happened, when all you want to do is get back to sleep. It sounds like quite a demanding job. With all the stuff thats been going on lately, Valar probably deserves a raise. Good work on fixing the game once again. --- Originally by: CCP Wrangler You're not supposed to feel like you're logging in to a happy, happy, fluffy, fluffy lala land filled with fun and adventures, thats what hello kitty online is for.
Amarria Black Clan Anthraxx	Posted - 2007.06.09 07:51:00 - [124] Originally by: CCP Valar Postmortem for tonights issues. Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration. This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues. This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs. The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure. The server was started back up but it quickly became apparent that this was still broken under load. At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again. When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan. However when the server was back up the same thing happened again. I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures. When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server. Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load. The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly. Signing off(and going back to bed) Ductoris, you keep pointing people here, and this last post from Valar (over an hour ago) is phrased in such a way as to imply the problem is fixed. It's not. The server may be up and running, but it's shakier than a 104 year old man with Parkinson's. Dock / undock times in the tens of seconds, autowarping to some random point 1mil KM from gate on system jump-in, jump-out lag in the tens of seconds, jump queues at 2-3 minutes for empty systems. So, can we get a REAL update?
Shalia Ripper Caldari High4Life Curse Alliance	Posted - 2007.06.09 07:55:00 - [125] Originally by: CCP Valar Postmortem for tonights issues. Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration. This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues. This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs. The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure. The server was started back up but it quickly became apparent that this was still broken under load. At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again. When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan. However when the server was back up the same thing happened again. I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures. When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server. Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load. The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly. Signing off(and going back to bed) Has anyone ever told you that you like like Billy Corrigan's zombie corpse? Oh, is the server up?
Ramius Decimus Caldari House Draconis	Posted - 2007.06.09 07:57:00 - [126] Edited by: Ramius Decimus on 09/06/2007 07:58:53 Originally by: Amarria Black Ductoris, you keep pointing people here, and this last post from Valar (over an hour ago) is phrased in such a way as to imply the problem is fixed. It's not. The server may be up and running, but it's shakier than a 104 year old man with Parkinson's. Dock / undock times in the tens of seconds, autowarping to some random point 1mil KM from gate on system jump-in, jump-out lag in the tens of seconds, jump queues at 2-3 minutes for empty systems. So, can we get a REAL update? That's strange, because I wasn't having any of the problems you were. And I was even in Jita. Must be all on your end.... or the node your system is on... Woot, server back up!!! Cheers, CCP.... ________________________________________________
Ramius Decimus Caldari House Draconis	Posted - 2007.06.09 08:02:00 - [127] Originally by: Siri Blue I come back from work...and am drunk...and WTF EVE IS DOWN!?!?!??!?! And I bet its some "oh, well, some admin forgot to clear some cash issue" so we d'oh...have to clear some and reboot...and crap"""" Get some people to do the daily maintenance who actually do the maintanance and don't just play minesweeper on the servers while they reboot!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! (111) You're allowed to drink at work?!? Wish I had a job like you... ________________________________________________
Amarria Black Clan Anthraxx	Posted - 2007.06.09 08:10:00 - [128] Originally by: Ramius Decimus Edited by: Ramius Decimus on 09/06/2007 07:58:53 Originally by: Amarria Black Ductoris, you keep pointing people here, and this last post from Valar (over an hour ago) is phrased in such a way as to imply the problem is fixed. It's not. The server may be up and running, but it's shakier than a 104 year old man with Parkinson's. Dock / undock times in the tens of seconds, autowarping to some random point 1mil KM from gate on system jump-in, jump-out lag in the tens of seconds, jump queues at 2-3 minutes for empty systems. So, can we get a REAL update? That's strange, because I wasn't having any of the problems you were. And I was even in Jita. Must be all on your end.... or the node your system is on... Woot, server back up!!! Cheers, CCP.... Logged out, rebooted my system, ran some connection tests. Pinging low, 1.5mb up and down consistently, no packetloss. It may just be isolated to the node(s) serving Tash-Murkon and Kador. Fun fun.
Frug Zenithal Harvest	Posted - 2007.06.09 08:12:00 - [129] It is this all-regions search of the contracts that's doing it I think. We wanted it, they didn't want to do it. We begged more. They gave in, and now look! At least we can search all regions :P - - - - - - - - - Do not use dotted lines - - - - - - - If you think I'm awesome, say BOOO BOOO!! - Ductoris Neat look what I found - Kreul Hey, my marbles
Templer Relleg Corp 1 Allstars	Posted - 2007.06.09 08:44:00 - [130] Edited by: Templer Relleg on 09/06/2007 08:46:00 Originally by: CCP Valar I called time of death for TQ as soon as I saw its status and a shutdown was initiated again. ... When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server. Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load. Dont even say "Death" and "TQ" in the very same post How long does it takes to do a index rebuilt and failover on a such beast? I remember some numbers about the database posted last year. And it isnt exactly a small database. Hopefully you had a backup image of the database before you began patching the server manually? Or are these done on-the-fly, or maybe during downtime? ps. thanks for such an extensive battle report. That was the most interresting i ever read! Corp 1 - Got what it takes?
parrowdox Caldari K-lash-nek-off	Posted - 2007.06.09 08:55:00 - [131] i was autopiloting, i got to the jita gate then everything froze for 5 mintues i try to log an it wydnt let me go to my character :( please sort it CCP we love u :)
maarud Coreli Corporation Corelum Syndicate	Posted - 2007.06.09 09:40:00 - [132] Originally by: Rauth Kivaro Temporary glue applied. Two in one day for maruud. Awesome, wish it was under better circumstances though Maarud. Proudly a Ex-BYDI member
Isan Danderoda	Posted - 2007.06.09 11:08:00 - [133] One thing I've grown to adore about Eve is that the Devs actually respond from time to time. Since I'm a DBA by daylight hours I really enjoyed this one. I can feel your pain Valar! I've done my share of late night flushes and compilations. Thanks for all the hard work!
Mr Bigglesworth Maza Nostra oooh Shiny	Posted - 2007.06.09 11:23:00 - [134] Originally by: parrowdox i was autopiloting, i got to the jita gate then everything froze for 5 mintues i try to log an it wydnt let me go to my character :( I'm pretty sure that part is working as intended
Helison Gallente Times of Ancar THE R0NIN	Posted - 2007.06.09 11:48:00 - [135] Please implement an emergency plan, that the server automatically stops (or restricts) systems like contracts or the market, if they are using too many ressources. It¦s much better if the contract-system is unusable for a short time, than the whole server goes down.
DvlDwg Gallente Merchant Miner Co-Operative	Posted - 2007.06.09 13:53:00 - [136] I'm not upset that the servers were down. I understand that servers going bad, data corrupting, somebody restarting something when they shouldn't is going to happen. What gripes me is the obvious lack of concern about NOTIFYING the player database and the, in my opinion, complete lack of unproffessionalism in locking the forums so no one can even read a message or post to get a time frame or hint of when things will be corrected. Bad form... very bad form. Post something from the get go. Notify your paying customers there is a problem, you expect it to be fixed by such and such time, and then post what the problem is after it's fixed. I'm not so addicted to this game that I won't stop giving you my $15 a month. If you don't care about me... I can assure you that I will quit caring about you very quickly. There is no hunting like the hunting of man..]
Korizan Oort Cloud Industries	Posted - 2007.06.09 16:36:00 - [137] Edited by: Korizan on 09/06/2007 16:42:26 Originally by: DvlDwg I'm not upset that the servers were down. I understand that servers going bad, data corrupting, somebody restarting something when they shouldn't is going to happen. What gripes me is the obvious lack of concern about NOTIFYING the player database and the, in my opinion, complete lack of unproffessionalism in locking the forums so no one can even read a message or post to get a time frame or hint of when things will be corrected. Bad form... very bad form. Post something from the get go. Notify your paying customers there is a problem, you expect it to be fixed by such and such time, and then post what the problem is after it's fixed. I'm not so addicted to this game that I won't stop giving you my $15 a month. If you don't care about me... I can assure you that I will quit caring about you very quickly. 1. Well when they are having database problems it effects the forums as well. This has been proven several times as the forums go off line during issues like these. 2. The servers were down and obviously CCP was working on it. I would rather they pay more attention to the work at hand then giving us hourly updates. I will tell you right now having people hover around or asking for updates while you are trying to fix something is annoying as HELL and is disruptive. So let CCP do there job and have a little patents. As soon as they find the problem and fix it they will let us know. Badgering CCP to give updates is NOT helping things instead you become part of the problem. And my $15 month goes to having a game on 23/7, so I say FIRST fix the issue and THEN let us know. END LINE
Cdr Foxbat Vanguard Frontiers INVICTUS.	Posted - 2007.06.09 16:51:00 - [138] Quote: Experiancing Database Issues, WTS Spellchecker Originally by: Marduk Felzhen Originally by: Vorian Atraties i bid my kidney... i bid his other kidney
brinelan Caldari Victory Not Vengeance Intrepid Crossing	Posted - 2007.06.09 16:59:00 - [139] Edited by: brinelan on 09/06/2007 16:59:07 Originally by: DvlDwg I'm not upset that the servers were down. I understand that servers going bad, data corrupting, somebody restarting something when they shouldn't is going to happen. What gripes me is the obvious lack of concern about NOTIFYING the player database and the, in my opinion, complete lack of unproffessionalism in locking the forums so no one can even read a message or post to get a time frame or hint of when things will be corrected. Bad form... very bad form. Post something from the get go. Notify your paying customers there is a problem, you expect it to be fixed by such and such time, and then post what the problem is after it's fixed. I'm not so addicted to this game that I won't stop giving you my $15 a month. If you don't care about me... I can assure you that I will quit caring about you very quickly. It has been said many many times on dev posts and dev blogs that the forum database is on the same machine as the tranquility database. A recent dev blog said they were going to be moving it so it is separate but has not been done yet. Put the torch and pitchfork away.. stuff happens. With these problems cropping up lately stemming from a change that they didn't seem to want to do until us users strongly requested it, I personally would rather see them remove the search feature where i can check other regions contracts, but thats just me. It will take more then a few downtimes to get me to stop paying my $30 / month. Ive played mmo's for about 8 years and this stuff happens. If anything, my eve experience with downtimes, and patches has been the best ive seen in all the mmo's i've played. -------------------------- Victory not Vengeance [VnV] Proud member of Intrepid Crossing [IRC]
Kleintje Pils Gallente	Posted - 2007.06.09 17:20:00 - [140] Originally by: brinelan Victory not Vengeance [VnV] VnV-Nation rockes !
Midou Li	Posted - 2007.06.09 17:54:00 - [141] Originally by: Kuseka Adama This wasnt just servers. This was the website/forums I think something a lot bigger than a simple glitch happened tonight guys... Its a raid! It's a DDos attack! Quick, to the internet mobile! I bet 4chan is behind it, they shutdown hal turner the same way! if so Quick call goonfleet, SA needs to raid 4chan STAT(even tho raiding 4chan is like **ing in an ocean of **..)
Elspeth Vigneron Caldari Phoenix Logistics Industries	Posted - 2007.06.09 19:41:00 - [142] Originally by: CCP Valar Postmortem for tonights issues. Around midnight the main contracts lookup database procedure got recompiled for some reason. Nothing unusual about that really. However, the query plan generated after that was not ideal and thus started non-selective contract lookups to take tens of minutes to run instead of the normal < 3 sec duration. This caused the database connection pools on nodes where players utilize the contracts system a lot to fill up with pending contract lookups and caused node deaths, lag and stuck issues. This also affected players not on those nodes because the database server went to 100% utilization on all of its CPUs. The server was taken down and the procedure that we had developed to address this after the last time this happened was implemented and testing while the server was down showed a pretty normal duration of the procedure. The server was started back up but it quickly became apparent that this was still broken under load. At this time, I got a call to take a look at this. I called time of death for TQ as soon as I saw its status and a shutdown was initiated again. When the server was down, I followed the same procedure as is standard in this case, but additionally flushed the procedure cache and forced it to select a proper query plan. However when the server was back up the same thing happened again. I tried fixing this live but its not much that we can do when the server is at 100% CPU load, so we decided that we had to take the server down yet again and do some drastic measures. When the server was down for the last time, fullscan stats were updated for the tables involved, indexes rebuilt and a failover initiated on the SQL Server. Then a query plan was created from the least selective parameters of the proc. This did fix the problem and the server is now under normal load. The procedure that was causing the issues was changed very recently to address this. However, as tonights incident shows, that fix did not help a lot. After reviewing the data collected tonight we believe that we can address this properly. Signing off(and going back to bed) I feel for you. Phoenix Logistics Industries 01010000010100100100010101010000010000010101001 00100010100100000010101000100111100100000010001 000100100101000101001011000010000
Nekuva The SMITE Brotherhood Curse Alliance	Posted - 2007.06.10 04:15:00 - [143] So uhh... HLW-HP, KLMT and the surrounding systems have been EXTREMELY laggy ever since the server problems last night. Any reason why? Server still going crazy? -_-_-_-_-_-_-_-_-_-_-_-_-_- Originally by: Haffrage One does not simply log in to Jita.

Ductoris Forum Moderator Interstellar Services Department	Posted - 2007.06.10 05:53:00 - [144] Un-Stickying somehow I can only think I'm cursing the server tonight...... forum rules \| Email us

Preston FateForger Caldari Caldari Provisions	Posted - 2007.06.10 06:58:00 - [145] Is the server still down? I still can't log in.

Pages: 1 2 3 4 [5] :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.