EVE Search - The current server issues

Check My IP Information

All Channels

EVE General Discussion

The current server issues

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 3 4 5 6 7 8 9 :: [one page]

Author	Thread Statistics \| Show CCP posts - 9 post(s)

GM Faolchu Game Masters	Posted - 2007.05.25 01:54:00 - [1] The server is currently experiencing database problems, the server team is looking into it. The server should be up and running shortly without any further issues. We apologise greatly for any inconvenience caused and hope that there will be no further issues.

T'Renn Crimson Squall Molotov Coalition	Posted - 2007.05.25 02:05:00 - [2] Aaaaaaand I hope my kid grows up to be executor of Billy Gates' will, but I know that's not going to happen. Eve has been super flaky lately. Get it sorted ffs - I can't get my emo on if I can't sit in dock. -- Wanna be a low-sec scumbag?
uNtOldPAIN Minmatar Blaine Institute for Advanced Research Apoapsis Multiversal Consortium	Posted - 2007.05.25 02:05:00 - [3] Thanks for the update..

CCP Admiral Chamrajnagar	Posted - 2007.05.25 02:09:00 - [4] Server is now online, we had a database problem where the database server went too 100% cpu and remained there for 1 hour. I made the decision after consulting with the GM's to take the server down gracefully instead of allowing it to melt down. A job had caused an abnormal load and the server was trying to catch up but could not. Upon startup of the cluster again a machine decided to reboot itsself and cause the infinite startup loop. After correcting that issue we are now online. My apologies for the delay in updating you.

Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 02:09:00 - [5] HEY OUR THREAD WAS FIRST NO FAIR MAKING YOUR OWN YOU WILL POST IN OUR THREAD
Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 02:15:00 - [6] Wow....way to ruin our fun...you guys locked our thread =/

Ductoris Forum Moderator Interstellar Services Department	Posted - 2007.05.25 02:17:00 - [7] Originally by: Dapanman1 Wow....way to ruin our fun...you guys locked our thread =/ GM threads usually take priority when deciding which thread will be the official thread. Had Faolchu not posted I would taking the first thread, which was admittedly yours. Now it's time to clean up GD so I'm locking the other threads, nothing personal. forum rules \| Email us

Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 02:19:00 - [8] /me thumbs nose at GMs
Kool Milds	Posted - 2007.05.25 02:21:00 - [9] it dont matter hwich post is which, im sure here in an hour or so we will be right back here watching the server crash over and over again..... /end rant

Ductoris Forum Moderator Interstellar Services Department	Posted - 2007.05.25 02:22:00 - [10] Originally by: Dapanman1 /me thumbs nose at GMs I don't like them much. They do nasty things to me on IRC. I can't tell you but I can show you on the doll...... forum rules \| Email us


Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 02:22:00 - [11] pst...thumb your nose at them

Ductoris Forum Moderator Interstellar Services Department	Posted - 2007.05.25 02:25:00 - [12] Originally by: Dapanman1 pst...thumb your nose at them Done. He didn't like it one bit. forum rules \| Email us

Tod Klemp Gallente K-Street Project	Posted - 2007.05.25 02:51:00 - [13] I think todays patch needs a rollback.
Morgain dVher Minmatar The Rat Pack	Posted - 2007.05.25 03:55:00 - [14] I still can't get on. Should I start getting paranoid? My tin foil had does have holes in it and I've been off my meds for a few days . . . or is it weeks? Picking up bottles to pay for my eve account takes up most of my day. M we do not, and never have tolarated to even a slight degree, any cheating within EVE. We are much too fully aware of the possible consequences of allowing any cheating to go on among the GM's.
Susitna Caldari	Posted - 2007.05.25 03:58:00 - [15] I dont think it is working. Status of server is unknowen
Morgain dVher Minmatar The Rat Pack	Posted - 2007.05.25 04:01:00 - [16] that's good to hear, normally my corpies make me take the meds. But I've run away from them. we do not, and never have tolarated to even a slight degree, any cheating within EVE. We are much too fully aware of the possible consequences of allowing any cheating to go on among the GM's.
Pelagiad Minmatar Project Gemini	Posted - 2007.05.25 04:01:00 - [17] And the server is down again or still???? Come on....EVE is 4 F'ing years old and its still in beta? Are you guys drunk or what. CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. I'd like to see a little less pictures of the Dev's drinking and a little more quality put into this product. You guy have an absolute great idea with EVE, bur for 4 years you've *ed away your chances to make it really shine through your lack of concern over stability and quality in favor of features and fluff. BTW who's God Dmn idea was it for the website/forums to share the same database with the game.....kinda makes for more lag and performance issues huh? [email protected]
Sentinel Eeex Caldari Lords Of Amber	Posted - 2007.05.25 04:03:00 - [18] Originally by: Tod Klemp I think todays patch needs a rollback. No. We need more features. We need more useless objects in space. We need new graphic engine, because we don't want to play the game, we want to watch it. After those changes, EVE can officialy be called "turn based game". It's almost there, just few bits and pieces missing...
Morgain dVher Minmatar The Rat Pack	Posted - 2007.05.25 04:03:00 - [19] Which proves I don't need them. tips another can in his rifter we do not, and never have tolarated to even a slight degree, any cheating within EVE. We are much too fully aware of the possible consequences of allowing any cheating to go on among the GM's.
Hamfast Gallente	Posted - 2007.05.25 04:04:00 - [20] reminds me of a boxing match... "he's down, but he's up again, and down again... no worries, 13 more rounds to go... make that 12 None of us is as dumb as all of us...

w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:05:00 - [21] Even the stickys broken Its not sticking! Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:06:00 - [22] Originally by: Pelagiad CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. No no no, the customers want FEATURES and CONTENT not quality... Originally by:** CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Morgain dVher Minmatar The Rat Pack	Posted - 2007.05.25 04:06:00 - [23] This "sticky" seems to have more post-it glue than super glue. It's falling down the forums. Up with you!! we do not, and never have tolarated to even a slight degree, any cheating within EVE. We are much too fully aware of the possible consequences of allowing any cheating to go on among the GM's.
BlazeRage Intentional Negligence	Posted - 2007.05.25 04:07:00 - [24] Its not sticking because the hamtser saliva glue factory melted along with the server.
Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:10:00 - [25] Quote: No no no, the customers want FEATURES and CONTENT not quality... I'm so very sorry, for a minute I forgot that my favorite part of any game is reading everything I could be doing if the game worked.
Micia Minmatar Thrace Inc Ushra'Khan	Posted - 2007.05.25 04:12:00 - [26] Originally by: T'Renn I can't get my emo on if I can't sit in dock. This is really funny, for some reason. _______ Oppressed? Enslaved? Stand up and fight back!
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:12:00 - [27] Originally by: Pelagiad I'm so very sorry, for a minute I forgot that my favorite part of any game is reading everything I could be doing if the game worked. You missed the sarcasm in my voice Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Hamfast Gallente	Posted - 2007.05.25 04:12:00 - [28] Originally by: BlazeRage Its not sticking because the hamtser saliva glue factory melted along with the server. Hey, stop picking on Hamster Glue None of us is as dumb as all of us...

Kreul Intentions Forum Moderator Interstellar Services Department	Posted - 2007.05.25 04:13:00 - [29] /me smacks the hamsters around a bit. Re-stuck! forum rules \| [email protected]

w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:13:00 - [30] Originally by: Hamfast Originally by: BlazeRage Its not sticking because the hamtser saliva glue factory melted along with the server. Hey, stop picking on Hamster Glue That wouldnt be funny... But your name makes it so Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.

Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:14:00 - [31] Originally by: w0rmy Originally by: Pelagiad I'm so very sorry, for a minute I forgot that my favorite part of any game is reading everything I could be doing if the game worked. You missed the sarcasm in my voice No I think you missed mine
Hamfast Gallente	Posted - 2007.05.25 04:15:00 - [32] Originally by: Kreul Intentions /me smacks the hamsters around a bit. Re-stuck! OUCH... I want Compensation... None of us is as dumb as all of us...
Judge Ment ECMI	Posted - 2007.05.25 04:15:00 - [33] Just come to say HELLO - Id like to sign in long enough to set my skill for a few days! Anyway to roll back this stupid patch? Maybe a few patches anyways
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:17:00 - [34] Originally by: Pelagiad Originally by: w0rmy Originally by: Pelagiad I'm so very sorry, for a minute I forgot that my favorite part of any game is reading everything I could be doing if the game worked. You missed the sarcasm in my voice No I think you missed mine Oh, in that case, I think ill go read the documentation on POS... Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Rhaegor Stormborn Sturmgrenadier Inc R i s e	Posted - 2007.05.25 04:17:00 - [35] 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second... 5.4.3.2.1.Less Than A Second...
D1ck Jones Old Detroit Crime Syndicate	Posted - 2007.05.25 04:17:00 - [36] Starting up...(0 sec.)
Jordon Spikes	Posted - 2007.05.25 04:18:00 - [37] free the hampsters, use the bunnies.
AirWalker Amarr Galactic Response Team	Posted - 2007.05.25 04:19:00 - [38] Originally by: Jordon Spikes free the hampsters, use the bunnies. i'd suggest using wolverines in heat.....they'd be hardcore
Janu Hull Caldari Order of Z Industries	Posted - 2007.05.25 04:19:00 - [39] Originally by: AirWalker Originally by: Jordon Spikes free the hampsters, use the bunnies. i'd suggest using wolverines in heat.....they'd be hardcore I thought they used them for CONCORD pilots. This is my sig, there are many others just like it. With me, my sig is worthless. Without (or with even) my sig, I am worthless...
Otuo	Posted - 2007.05.25 04:20:00 - [40] So if we had a short skill running when the lag started and could not get it changed to a long one are you going to reimbure us for lost training time because i've been up and staying up so i can switch the skill. All this just to fly a drake :(

hydraSlav Synergy Evolved Sparta Alliance	Posted - 2007.05.25 04:20:00 - [41] Originally by: Jordon Spikes free the hampsters, use the bunnies. Thanks! While cursing everything cause the server issues started right as my skill finished and i didnt get a chance to start a new one, this quote for some reason made me LOL in RL == Above comments are my personal views Oveur >Local shouldn't be a tactical tool, it's for chat
SpyTech	Posted - 2007.05.25 04:20:00 - [42] Originally by: Judge Ment Just come to say HELLO - Id like to sign in long enough to set my skill for a few days! Anyway to roll back this stupid patch? Maybe a few patches anyways Hmm a roll back might be in order but as long as it is rolled back to my long skill training as I NEED sleep!!!! I agree, just let the servers get up long enough for me to log in, change my skill to a few days as well so I can stop worrying about it and I can get some sleep!! Please :) ShyTech
Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:20:00 - [43] Originally by: w0rmy Oh, in that case, I think ill go read the documentation on POS... Now that's the spirit, but there is a problem...I don't think the documentation is upto date on the eve-online web site. BTW my new favorite part of EVE(Besides actually playing) is the anticipation I get watching the following: Status:Starting Up(4,3,2,1) yes finally Status:Starting Up(4, 3, 2, 1) Ok this time for sure Status:Starting Up(4, 3, 2, 1) Can I sue for emotional abuse?
Jordon Spikes	Posted - 2007.05.25 04:20:00 - [44] Originally by: AirWalker Originally by: Jordon Spikes free the hampsters, use the bunnies. i'd suggest using wolverines in heat.....they'd be hardcore And i wonder how you know this.
QwaarJet Gallente hirr Morsus Mihi	Posted - 2007.05.25 04:21:00 - [45] I'm not even bothered this time. This is what i've come to expect from CCP tbh. "Hobbes, she stepped into the Perimeter Of Wisdom.Run!"
firepup82 Caldari Provisions	Posted - 2007.05.25 04:21:00 - [46] F this! NN all. Gone to bed.
AirWalker Amarr Galactic Response Team	Posted - 2007.05.25 04:21:00 - [47] Originally by: Jordon Spikes Originally by: AirWalker Originally by: Jordon Spikes free the hampsters, use the bunnies. i'd suggest using wolverines in heat.....they'd be hardcore And i wonder how you know this. a gentleman never tells
Xeth Dunai	Posted - 2007.05.25 04:24:00 - [48] You know, despite how frustrating this is, I have to admit, it's funny. I actually was trying EVE because WoW was having huge server bugs the last couple weeks, and got tired of getting DCed. Sorry, people! The crash bangs are following me...
Rooker Lysian Enterprises United Corporations Against Macros	Posted - 2007.05.25 04:25:00 - [49] Edited by: Rooker on 25/05/2007 04:25:31 Quote: A job had caused an abnormal load and the server was trying to catch up but could not. Upon startup of the cluster again a machine decided to reboot itsself and cause the infinite startup loop. Sorry man. I was trying to divide the number of my war targets who have logged on since it went active with the amount of ammo I have in a certain hangar.... and well, since NONE have logged on in 3 days.... Damn computers.. just can't cope with dividing by zero.. :(
Erikel Cosmic Odyssey YouWhat	Posted - 2007.05.25 04:26:00 - [50] Edited by: Erikel on 25/05/2007 04:27:28 Originally by: Pelagiad And the server is down again or still???? Come on....EVE is 4 F'ing years old and its still in beta? Are you guys drunk or what. CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. I'd like to see a little less pictures of the Dev's drinking and a little more quality put into this product. You guy have an absolute great idea with EVE, bur for 4 years you've *ed away your chances to make it really shine through your lack of concern over stability and quality in favor of features and fluff. BTW who's God Dmn idea was it for the website/forums to share the same database with the game.....kinda makes for more lag and performance issues huh? Now i know why they took the server down a second time. They are messing with that arsehat.

Gjs312 Caldari Black Lance NBSI Alliance	Posted - 2007.05.25 04:27:00 - [51] Any update? Not usually one to complain.. but it has been pretty bad today.
Loyal Servant Caldari The Short Bus Squad The SUdden Death Squad	Posted - 2007.05.25 04:27:00 - [52] I wonder how many new humans are being conceived tonight? There is an urban legend that back when there was a major power outage in the 1960's or something here in the US - there was a ton of new births 9 months later TSBS - Eve's Premier podding service!
Edgard Clochard Iron Shield Associates	Posted - 2007.05.25 04:27:00 - [53] Tonight was supposed to be "Stay Up All Night Ratting Night" for me. I guess I'll just go outside and stare at the stars for a while.
Johnny M	Posted - 2007.05.25 04:29:00 - [54] Originally by: Otuo So if we had a short skill running when the lag started and could not get it changed to a long one are you going to reimbure us for lost training time because i've been up and staying up so i can switch the skill. All this just to fly a drake :( God what a nice thought, hmm CCP any comment on this
Jesters Knight Eclipse Enterprises Empire Research	Posted - 2007.05.25 04:30:00 - [55] Originally by: w0rmy Originally by: Pelagiad Originally by: w0rmy Originally by: Pelagiad I'm so very sorry, for a minute I forgot that my favorite part of any game is reading everything I could be doing if the game worked. You missed the sarcasm in my voice No I think you missed mine Oh, in that case, I think ill go read the documentation on POS... ROFL dont get me started on POS's Highsec POS labs with no risk of scamming
SpyTech	Posted - 2007.05.25 04:30:00 - [56] Starting up...(0 sec.) I don't think they will be getting the servers up and running right again till tomorrow (way later today in some time zones) /me is soo tired and wants to pod a polaris ship.. ShyTech / SpyTech / TooTired2GoOn
Siigari Kitawa Gallente The Aduro Protocol	Posted - 2007.05.25 04:31:00 - [57] Can't.. play.. eve... _________________ Burn.
Jesters Knight Eclipse Enterprises Empire Research	Posted - 2007.05.25 04:31:00 - [58] dangit, my girlfriend just left, and i want to set long skills so i can go to bed >.< Highsec POS labs with no risk of scamming
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:32:00 - [59] Originally by: Jesters Knight Originally by: w0rmy Oh, in that case, I think ill go read the documentation on POS... ROFL dont get me started on POS's Glad you got the joke Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Hamfast Gallente	Posted - 2007.05.25 04:33:00 - [60] Originally by: Johnny M Originally by: Otuo So if we had a short skill running when the lag started and could not get it changed to a long one are you going to reimbure us for lost training time because i've been up and staying up so i can switch the skill. All this just to fly a drake :( God what a nice thought, hmm CCP any comment on this They warned us Monday to set a long skill to train... Ok, So I should have gone for Drone Interfacing 5... None of us is as dumb as all of us...

Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:34:00 - [61] Originally by: Erikel Edited by: Erikel on 25/05/2007 04:27:28 Originally by: Pelagiad And the server is down again or still???? Come on....EVE is 4 F'ing years old and its still in beta? Are you guys drunk or what. CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. I'd like to see a little less pictures of the Dev's drinking and a little more quality put into this product. You guy have an absolute great idea with EVE, bur for 4 years you've *ed away your chances to make it really shine through your lack of concern over stability and quality in favor of features and fluff. BTW who's God Dmn idea was it for the website/forums to share the same database with the game.....kinda makes for more lag and performance issues huh? Now i know why they took the server down a second time. They are messing with that arsehat. So you like the way the game is working right now then? Just being clear, when would you say was the last time Eve was working the way it's supposed to be working, as communicated to us by CCP?
aquontium Gallente Fourth Circle	Posted - 2007.05.25 04:35:00 - [62] DI V? I was gonna train ADI V! ..........but then I remembered I needed the last high slot for something else so I set a shorter skill.
Talidorn Cutting Edge Incorporated RAZOR Alliance	Posted - 2007.05.25 04:37:00 - [63] I've offered this during other "issues" that CCP has had like this... I am an Enterprise Monitoring & Management consultant. Care to make me a full-time CCP employee? I'd be glad to head up a new team at CCP that will catch these issues and alert you well before you have such issues. My product speciality is HP OpenView. OpenView Transaction Analyser (OVTA) would help you identify these issues, bottlenecks, and poor response servers. The benefit of hiring me would be that I would be your new whipping boy... "Blame the monitoring guys!" LOL Let me know if you are interested. Talidorn
Xeth Dunai	Posted - 2007.05.25 04:37:00 - [64] It's kinda sad when "Status: Unknown" makes me grin like a fool.
Gibb0	Posted - 2007.05.25 04:37:00 - [65] Reaches for LotR Online Any kind of update from anyone in power would be hot.
Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 04:38:00 - [66] So as previously stated in this thread, ahem I WANNA SWITCH MAH SKILLZ you think they'll give us some sp's as compensation? Maybe extend everyones subscriptions...
pretezels	Posted - 2007.05.25 04:39:00 - [67] The hamster is dead-quick run to the pet store.
Celduil	Posted - 2007.05.25 04:39:00 - [68] Originally by: Loyal Servant I wonder how many new humans are being conceived tonight? There is an urban legend that back when there was a major power outage in the 1960's or something here in the US - there was a ton of new births 9 months later Answer: None, Nerds don't have girlfriends. :)
Hamfast Gallente	Posted - 2007.05.25 04:41:00 - [69] Originally by: pretezels The hamster is dead-quick run to the pet store. I am not dead... I just type slow None of us is as dumb as all of us...
Siri Blue Gallente Quantar Swords SynchronizerZ	Posted - 2007.05.25 04:41:00 - [70] Ok, now you made me listen to the live dev blog thing! Wake those CCP dudes in London and Iceland up already! Its late enough! Spank those Hamsters!!!

Bam1 Gallente Infinitus Odium	Posted - 2007.05.25 04:41:00 - [71] Does anyone else still get a message saying Server Unknown? \|Bug Reporting\|Patch Notes\|
RuleoftheBone Minmatar Veto. Academy Veto Corp	Posted - 2007.05.25 04:43:00 - [72] Originally by: Bam1 Does anyone else still get a message saying Server Unknown? Yes @ 0540 BST "Lead Me..Follow Me..Or get the **** out of my way" General George Patton USA
invaderzim Deep Core Mining Inc.	Posted - 2007.05.25 04:43:00 - [73] Yes this is really annoying and should be fixed... rapidly and permanantly. But seriously. How long have we been playing this game? Most of you KNOW to set long skills in advance and for a few days following a patch. There are ALWAYS problems on and after patch day. It's not how it should be, but it is how it has always been. ----------------- "Oh, he's very popular Ed. The sportos, the motorheads, geeks, **s, bloods, waistoids, dweebies, **heads - they all adore him. They think he's a righteous dude."
Siri Blue Gallente Quantar Swords SynchronizerZ	Posted - 2007.05.25 04:43:00 - [74] The same thing as last night...node deaths, lag, crashes, lag and lag... Seems a more severe problem than just "need to reboot"
Isaaq	Posted - 2007.05.25 04:44:00 - [75] Edited by: Isaaq on 25/05/2007 04:43:18 Status: Starting Up drumroll (again)
Gjs312 Caldari Black Lance NBSI Alliance	Posted - 2007.05.25 04:44:00 - [76] up in 10 mins it seems.. Small print: 10 ccp minutes may or may not have any relation to real minutes. No refunds, **** you we already have your money.
UGWidowmaker Caldari Setenta Corp Xelas Alliance	Posted - 2007.05.25 04:45:00 - [77] yes. unknown here is ccp having favoritism on the forums ?
Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:45:00 - [78] Originally by: invaderzim Yes this is really annoying and should be fixed... rapidly and permanantly. But seriously. How long have we been playing this game? Most of you KNOW to set long skills in advance and for a few days following a patch. There are ALWAYS problems on and after patch day. It's not how it should be, but it is how it has always been. Imagine what could be done if people would not pull out the excuse of "well that's just the way its always been". And demand more, expect better.
Ribijk	Posted - 2007.05.25 04:51:00 - [79] -.-, its not going to work.
Gibb0	Posted - 2007.05.25 04:52:00 - [80] Good luck everyone 10 secs :)

Dapanman1 Amarr Beets and Gravy Syndicate	Posted - 2007.05.25 04:53:00 - [81] Now we all log in and crash it again...im ready for the deja vu
Gibb0	Posted - 2007.05.25 04:53:00 - [82] Yey!
Ribijk	Posted - 2007.05.25 04:53:00 - [83] Good work so far ccp, try to make it last more than 4 1/2 minutes this time please thanks.
Morgain dVher Minmatar The Rat Pack	Posted - 2007.05.25 04:53:00 - [84] Imagine Eve fixed. http://www.youtube.com/p.swf?video_id=jEOkxRLzBf0&eurl=http%3A//www.google.com/search%3Fsourceid%3Dnavclient%26aq%3Dt%26ie%3DUTF-8%26rls%3DGGLJ%2CGGLJ%3A2006-48%2CGGLJ%3Aen%26q%3DJohn&iurl=http%3A//img.youtube.com/vi/jEOkxRLzBf0/2.jpg&t=OEgsToPDskJtYJajjmHNoLCbr2bG8EH5 we do not, and never have tolarated to even a slight degree, any cheating within EVE. We are much too fully aware of the possible consequences of allowing any cheating to go on among the GM's.
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:54:00 - [85] Originally by: invaderzim I'm not excusing anything. Everyone here has a legit gripe. That shouldnt stop anyone from excercising a little forethought with regards to skill training on patchday. edit: punctuation Problem is, everything believed a few hours ago (Prior to the issues) that the patch had been deployed sucessfully, so they switched back to the 5hours skills they had to train... You cant blame the users in this situation. Had the deployment not gone well, and taken 24hours, then id agree with you. Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Kasriel	Posted - 2007.05.25 04:55:00 - [86] is anybody else having issues with actually entering the game? i've got no problems logging in, no problem selecing a character but i go to enter the game and it just hangs..
Jonathos	Posted - 2007.05.25 04:57:00 - [87] I'm in
Pelagiad Minmatar Project Gemini EternalRising	Posted - 2007.05.25 04:57:00 - [88] Originally by: Kasriel is anybody else having issues with actually entering the game? i've got no problems logging in, no problem selecing a character but i go to enter the game and it just hangs.. Give it time. It's alot like Windows, even tho it looks like its up and ready , its not really up and ready.
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 04:58:00 - [89] Originally by: Kasriel is anybody else having issues with actually entering the game? i've got no problems logging in, no problem selecing a character but i go to enter the game and it just hangs.. 20,000 people trying to log in at once.. give it a minute! Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Jonathos	Posted - 2007.05.25 05:00:00 - [90] I can't right click in open space...anyone else havin that issue?

Rooker Lysian Enterprises United Corporations Against Macros	Posted - 2007.05.25 05:06:00 - [91] Edited by: Rooker on 25/05/2007 05:04:50 Originally by: Morgain dVher big long ass huge monstrous freaking URL zomg.... you broke the damn forum :\
Pang Grohl Gallente	Posted - 2007.05.25 05:06:00 - [92] Am sad because the status readout said zero (0) online, but it lied. I didn't get EVE all to myself tonight. Si non adjuvas, noces (If you're not helping, you're hurting)
Almarin Enchura	Posted - 2007.05.25 05:32:00 - [93] CCP! I must protest. These wild parties in the server room are not good for EVE. I mean, last time you tripped over the extension cord, and this time you've gotten the servers drunk. This has to stop!
Franga Caldari NQX Logistics	Posted - 2007.05.25 07:51:00 - [94] So - how about those server issues? That first post was 4 hours before now. I come to log on and it would appear as though things are still iffy. Any update on what's doing? Kthxbye! _____________________________ Eldo spanked my sig but I can't be bothered changing it just now.
Cynic Tool	Posted - 2007.05.25 07:54:00 - [95] They are currently rebooting the server then HOPEFULLY everything will be good as new /me crosses fingers =D
lordopic	Posted - 2007.05.25 07:54:00 - [96] Originally by: Franga So - how about those server issues? That first post was 4 hours before now. I come to log on and it would appear as though things are still iffy. Any update on what's doing? Kthxbye! So its not just me with sleep in my eyes hitting the wrong keys, the server is still down then?
charles laforge	Posted - 2007.05.25 07:57:00 - [97] well 9am uk time and it looks as if were still offline, all i get is checking status when i try to log in anyone else having issues?
w0rmy M. Corp M. PIRE	Posted - 2007.05.25 07:58:00 - [98] Originally by: lordopic So its not just me with sleep in my eyes hitting the wrong keys, the server is still down then? Edit just read the post above which im sure wasnt there a mo ago. No, it came up... then went down, then came up again, then... Well you get the idea. Originally by: CCP Oveur I'm very sorry w0rmy, I beg your forgiveness.
Iva Soreass FireStar Inc	Posted - 2007.05.25 07:59:00 - [99] Same , i only wanna change skill ffs. Hurry up n fix it!. Originally by: Farscape Hw,GoonSwarm* Member* im not a goon Evolution
Juil Gallente Trade and Research Technologies	Posted - 2007.05.25 08:00:00 - [100] Ok... down again, who the hell swapped the Hampsters for Mice? Don't do it, mice have smaller feet and bodies there for the wheel don't turn as fast and the server doesn't run as well.. Put the Hamsters BACK in the cage, then give them some Redbull. Or even better, Upgrade the hamsters to Ferrits who run faster, Eat Hamsters and Rabbits and work ten times better.. but of course that would require a hardware update for bigger wheels... - One very bored Juil - Juil Intergrated Information Services Information is Power

Tsavong Lah Caldari Setenta Corp Xelas Alliance	Posted - 2007.05.25 08:02:00 - [101] lol, hope it wasn't the patch.. if it was, grats I got a jar of dirt!
Cynic Tool	Posted - 2007.05.25 08:04:00 - [102] Edited by: Cynic Tool on 25/05/2007 08:03:52 Everything WAS up and running for a bit, but like all the other patches that were deployed before it, we had the server coming back online issue. then the immense lag issue after about an hour or so of play, now we are at the reboot everything part of the patch deployments. EDIT: There we go, its coming back up now!
lordopic	Posted - 2007.05.25 08:04:00 - [103] 8-mins
Lordstaysin	Posted - 2007.05.25 08:18:00 - [104] Its up
Lori Carlyle LuthorCorp Combat Division	Posted - 2007.05.25 08:18:00 - [105] Originally by: Tobias Sjodin Just shut the servers down and put everyone's subscriptions on pause until you can fix the server and the server lag tbh. I kinda like this idea. ----------------- FIX THE FREAKING DRONES ALREADY
Kali Cortez coracao ardente Triumvirate.	Posted - 2007.05.25 08:20:00 - [106] Originally by: Tobias Sjodin Just shut the servers down and put everyone's subscriptions on pause until you can fix the server and the server lag tbh. lol just cancel your subscription I'll call you when its fixed flavor

Huitzilopochtli Tlaloc Forum Moderator Interstellar Services Department	Posted - 2007.05.25 08:26:00 - [107] Server is back up and running. Im sure there will be a official announcement about the problems at some point today. - Thanks Hutch. ____ forum rules \| [email protected] \| ME They call me Hutch. I have forgotten why


CCP Valar	Posted - 2007.05.25 08:47:00 - [108] A quick post mortem after this horrible night. The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables. The SQL Server is now back to its normal self. The development team will hopefully have a proper fix for this ready soon. And now back to bed ------ Valar Database admin - Server operations team CCP Games How to write a good bugreport

Nezz Jaran	Posted - 2007.05.25 08:59:00 - [109] Edited by: Nezz Jaran on 25/05/2007 09:00:20 Cool, so the server farm is "now fixed" and it's time for me to go to bed (1 am). Glad I had a chance to ******* play tonight. Don't like playing when the servers are bouncing around, too much of a chance for something to go wrong. And out of curiousity, when you say "SQL Server" you are speaking in generic terms, right? You're not actually talking MS SQL Server? Cuz if you mean MS SQL Server, I think I just found your problem (it's all about Oracle on Unix).
Emeline Cabernet Amarr KVA Noble Inc.	Posted - 2007.05.25 09:00:00 - [110] /me slips into thread.. i see you all been having fun /me goes to find BREE!

Marasume	Posted - 2007.05.25 09:15:00 - [111] Originally by: Nezz Jaran And out of curiousity, when you say "SQL Server" you are speaking in generic terms, right? You're not actually talking MS SQL Server? Cuz if you mean MS SQL Server, I think I just found your problem (it's all about Oracle on Unix). If you switch to oracle and need a DBA give me a call :P
Vergil 577	Posted - 2007.05.25 09:15:00 - [112] Edited by: Vergil 577 on 25/05/2007 09:14:48 Wait a CPU overload? Imagine how much that would have to take? My system would completly blow up before that... Well glad the problems fixed now we can all go back to our non-lives and have fun (including me). <Bad Spelling Day>
Ralitge boyter Minmatar Mordus Angels	Posted - 2007.05.25 09:36:00 - [113] Originally by: Nezz Jaran Edited by: Nezz Jaran on 25/05/2007 09:00:20 Cool, so the server farm is "now fixed" and it's time for me to go to bed (1 am). Glad I had a chance to ******* play tonight. Don't like playing when the servers are bouncing around, too much of a chance for something to go wrong. And out of curiousity, when you say "SQL Server" you are speaking in generic terms, right? You're not actually talking MS SQL Server? Cuz if you mean MS SQL Server, I think I just found your problem (it's all about Oracle on Unix). They mean MS SQL server all right the entire thing (EVE Cluster) is windows based. Please don't start screaming how bad it is we all know but then again I am sure any and every other system has problems of it's own CCP choose M$ for a good reason and have build a quite sucessful business using their software. So unless you have started build and are still running a bigger and more sucsesful business on other software which you can prove is the reason for you being more sucesful then them please please please stop the M$ sucks rant and accept that M$ can actualy preform good enough to run a super computer hosting the biggest virtual universe in the world. (just my two cents) ------------------------------------------- Should you disagree with me, well I guess that is because I disagree with you. If you have a problem with that please feel free not to tell me.
Viktor VonCarstein Amarr Phoenix Industries	Posted - 2007.05.25 09:37:00 - [114] Thanks for the Update Valar. Seems like a busy night for you These things happen I guess, keep up the good work. Do you get a late start in the morning if you were up all night? http://hell-raisers.xippy.co.uk
Thodoros Minmatar Mythos Corp RAZOR Alliance	Posted - 2007.05.25 09:40:00 - [115] SQL server??? Now you cant be serious about it? If thats true now we know why we have so much lag all the time. No wonder the server cant handle big fights with over 100 peers on the same system. Very sad.
Pallidum Treponema KIA Corp KIA Alliance	Posted - 2007.05.25 09:47:00 - [116] Being a rabid unix engineer, I have to say that if it works, don't mess with it. By the sound of it, an oracle cluster wouldn't have done much better. A misconstructed query (or queries rather) would've killed any database (subject to setup). Anyway, to the CCP team, keep up the good work. Speaking as someone who's put in almost 60 hours so far this week, I know what it's like to work long nights. :)
Stamford Bridge	Posted - 2007.05.25 09:52:00 - [117] passes the cheese around for all the whiners "ccp suck" "ms sql suck" "blah blah suck" really thought i stumbled upon the WoW forums. Annoying, yes it is when TQ has a shtfit, but at least we are kept in the loop. most companies blame isp or your pc's (sideways glance at EA and Sony et al)
Qual Gallente Cornexant Research	Posted - 2007.05.25 09:54:00 - [118] Originally by: Thodoros SQL server??? Now you cant be serious about it? If thats true now we know why we have so much lag all the time. No wonder the server cant handle big fights with over 100 peers on the same system. Very sad. WHat are you talking about? Perfomance differnce on MS SQL and Oracle is minimal at this point in time. When you have a non run of the mill system like this, its really all about how well the guys who writes quesries can work around limitations of relational databases, and not so much about which database engine you are using. "The short version: Qual is right." -Papa Smurf
BCBArclight Odessa Operations Sylph Alliance	Posted - 2007.05.25 09:57:00 - [119] Originally by: CCP Valar This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. Good old SQL server, if you need someone else to throw stuff at when it breaks I need a job after finishing uni now I worked in SQL support for a year, I have lots of ways to mess it up that havent been tried yet Odessa Operations are Recruiting
Pallidum Treponema KIA Corp KIA Alliance	Posted - 2007.05.25 10:14:00 - [120] I find it amusing how people go "SQL Server? That sucks!" I think one of the problems is presentation. When CCP says "SQL Server", a lot of people think about the old overloaded P3 machine they've got at work, and decide that since the SQL Server at work is crap, CCPs must be too. Now, if I'm not entirely mistaken, "SQL Server" in this case actually refers to a fairly large cluster with several individual high-end servers. All of those machines are multi-processor and possibly multi-core as well, with maxed memory (16GB or more each), attached to a SAN (Storage Area Network). In other words, as far away from the old P3 desktop machine as you can get. Still, it gets presented as a "server", which makes people think of the one single machine they're used to. If CCP instead decides to present things as the "Eve cluster" or the "SQL Cluster", people might get a better idea of what they're talking about. Or, described another way: Your SQL server at work? That's the rifter over there, don't break it. CCP's SQL cluster? It's the battleship blob that just ganked you. Oh, and they're setup for remote repping too.

Xrious	Posted - 2007.05.25 10:15:00 - [121] Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? I apologise and know that you need sleep, but employ someone else. Thats really what we're paying for. We all abuse M$, but I had alot less problems with SP2 than CCP seems have with their own systems. Sorry, just had the message pop up on my screen telling me that the server is shutting down at 11am for its normal reboot. Very very anoying when you've spent hours getting in. Kinda rubbing salt in the wounds there CCP. X Sig? what sig?
BCBArclight Odessa Operations Sylph Alliance	Posted - 2007.05.25 10:21:00 - [122] Originally by: Xrious Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? Not too uncommon, some companies have procedures that need to be followed when certain conditions arise. These need to be done asap to either save data or minimise effect on other systems, by whoever is there at the time. Its possible this was the case or the junior DBA wanted a laugh and pressed the big red button Odessa Operations are Recruiting
Mortim Minmatar Madison Industrial Co. Sylph Alliance	Posted - 2007.05.25 10:24:00 - [123] Aside from any complaints that have already been fronted, I'd like to do two things. Firstly, offer my thanks, EvE Online is nothing short of genius, the fact that the demands, and usage is wild, unstable, and always looking to change can't make keeping anything like this stable easy in the slightest. The hour maintenance time is far better than the constant problems that other, less innovative games suffer. My salute to your efforts over, my second point was that you always offer some form of explanation. Numerous people would be content with a simple "It's working", but the involvement with players (of which Im guessing a good number are of a computing nature) is great. So my question - Is there any way in which we can be more involved with the problems encountered, disasters averted and thought that goes into maintaining and upgrading EvE at all? A server team blog or some such would be ideal (pretty sure one doesn't exist). I'm personally starting my career in Systems Development, and insight into the backbone of something Im very familiar with would probably prove to me it's not all "magic". Mort
Devious	Posted - 2007.05.25 10:37:00 - [124] How many games have you played that are 3-4 years past beta where the dev's and support staff tell you what went wrong and what was fixed or will be fixed??? i can count only one. EVE Online.
Ifni Applied Eugenics	Posted - 2007.05.25 10:37:00 - [125] I love the way loads od people criticise CCP for their choices So far mud has been slung at CCP, Microsoft, the junior DB admin, the hardware, and at each other. The people at CCP know their setup better than anyone else, and when a problem like todays arises, there are certain procedures in place to deal with it. It is only in exceptional circumstances that a problem gets elevated to the point where you wake up the guy above you in the food chain, as can be seen by tonight's activities. Yes, it sucks, yes, you feel hard done by, but the EULA that you all agreed to states that the performance of the game may vary. As has been said before certain problems simply can not be foreseen prior to a complete roll out. Hell, why do you think Microsoft, the largest OS manufacturer in the world, rolls out constant updates and patches? And they've been doing it for decades. So quit your moaning, and rather than belittle the guys for doing their jobs, congratulate them for keeping the server running for the other 99.9% of the time. <3 Valar, <3 junior DB admin. You take what is offered. And that must sometimes be enough.
Marakish Kalar Minmatar Blancmange Interstellar Kilts Archaean Cooperative	Posted - 2007.05.25 10:42:00 - [126] Edited by: Marakish Kalar on 25/05/2007 10:43:08 Well done guys good to see the duty admin did what they could, but then decided to call out the guy who gets paid millions to do his job.... When I was younger this whole episode would have had me pulling my hair out complaining and B B Biting my tonge. I'm glad to say that now it doesn't, does? d..? What was I saying? zzzzzz Keep up the good work guys, and thanks. . Marakish Kalar [BLANC]MINMATAR'S EDGE
Xrious	Posted - 2007.05.25 10:54:00 - [127] Edited by: Xrious on 25/05/2007 10:54:06 I'm sorry, I'm not aiming to CCP bash, but the stability of the system isn't good enough IMHO. SQL server's don't need to be rebooted everyday, they stay up for years, not 23 hours. As someone else pointed out, running the forums and the website on the same cluster as the game? great, game server goes down, the players can't even find out about it. Someone said that how often does a company give explanation's on why a system screwed up 3-4 years out of beta, no offense but a system shouldn't be having these kinds of problems 3-4 years out of beta. Giving explanation's on what has happened is great, we really do appreciate it, but I think the player base demands it. Eve is an intelligent, strategic game, not a reflex FPS and the type of player tends to be more mature (well, mostly ) that you'd find on other games out there. A vast percentage of the player base understands the technology behind Eve so they could hardly try to cover it up. IMHO if there was an alternative to Eve out there, then I'm sure things would be different. People would have a choice, people would vote with their wallet. As it is, Eve is a one off, its unique so there's isn't anywhere else we can go for our 'fix', but it doesn't mean we need to be quiet about it. Anyway, just my 2 cents. Hopefully people at CCP are reading this. If nothing else gets done, could we at least have more information when things like this happen? 4 hours ( I think I read) bewteen updates is perhaps a little slack. Keeping the user's informed costs you little, but makes a massive different to us. Thanks X Edit: typo
Ethaet	Posted - 2007.05.25 10:55:00 - [128] Originally by: BCBArclight Originally by: CCP Valar This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. Good old SQL server, if you need someone else to throw stuff at when it breaks I need a job after finishing uni now I worked in SQL support for a year, I have lots of ways to mess it up that havent been tried yet SQL servers mess themselves up without need for a human fairly regularly :P :D
Rekindle	Posted - 2007.05.25 10:58:00 - [129] so many back seat admins. Kinda late in the design to be whining about SQL isn't it? (besides SQL is a great product). -------------------------------------------
Digital Anarchist	Posted - 2007.05.25 11:01:00 - [130] Originally by: CCP Valar The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. Dear CCP, please don't nerf the global view of contracts because of this. ---------------- Nerf government!

Suzii	Posted - 2007.05.25 11:02:00 - [131] are guristas supposed to drop only ammo 90% of the time
Aleksandr Cirtus Caldari Stronghold corp Curse Alliance	Posted - 2007.05.25 11:08:00 - [132] Edited by: Aleksandr Cirtus on 25/05/2007 11:07:21 Personally, I would like to thank CCP and the dev team, and the GM team, all at the same time. True, the issues were starting to get annoying, but they identified and rectified the problem in short order. In addition, a GM was able to assist with a side effect in one of my missions within a minute. A MINUTE. EVE has come a long way in four years, and I would like to see all the nay-sayers pause and reflect on past experiences and dare to tell me things are getting worse. Cheers to all! Aleksandr Cirtus Senior Knight and Ghost Fleet Pilot of Stronghold Curse Alliance
Oratu Brutor tribe	Posted - 2007.05.25 11:08:00 - [133] lol, nerds. Pre deployment testing ftw !!!! Doh! iam a nerd too :(
Tiger Ma	Posted - 2007.05.25 11:15:00 - [134] I run a database consulting company and basically make my money fixing databases and making them go faster. You're probably now expecting me to start ranting about MS SQL Server, junior DBAs and OMG where's my compensation for lost training time. Er, no. I do have to wonder if a clustered MS SQL Server farm is the best way to go, not out of any dislike of MS SQL Server (does a great job when used appropriately; I know large banks that use it very successfully for systems that with respect are considerably larger and more complex than EVE)... but because of the hardware constraints any MS architecture imposes. What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. But I can't see CCP being willing to walk away from some serious hardware investments to change their fundamental architecture. (Although think about solid state devices for all your tempdbs; it gets you past the 16Gb memory limit without architectural changes.) SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. An obvious win CCP are already discussing in public is separating the web database from the game database. That's really a no-brainer and it will have disproportionally good effects as the main DB won't have to cache web-related data or query plans anymore. I'd suggest a third database (on dedicated hardware) for character info is a simple next step. I'd even wonder how much of the game database could be distributed... the contracts system is pretty self-contained, for example, with nothing outside contracts obviously depending on it. Anyway, CCP, you have some good talent in your player base. I and many others would be absolutely delighted to volunteer under any reasonable NDA and T&C. If I'm allowed to use you as a reference I'll do it for free. 8-) Lastly, I've totally been the one who had to bring down a production environment on no notice. I understand why it has to happen. Good call on taking an unpopular step to get the job done. (And let me know if you want a hand query tuning!)
Armoured C	Posted - 2007.05.25 11:15:00 - [135] Edited by: Armoured C on 25/05/2007 11:15:38 great work on getting the problem sorted to all those moaning at CCP and dev team and server admin and what ever here a question for you try running your own server capabable of running 30k pilots in one go then we shall see how much we could moan at you every other game you have to log into a different server to do either missions , pvp and what not this is the only mmo i know of where it is all intergrated so shut up with you moaning do you kniw everything about there servers or anything about there set up at all? have you managed a 30k plus pilot server with intergrated website and forum? if any of the awnser are no and you still want to moan just shut up, they are dong the best they can in any situation you get my personal thanks CCP it could of been worst and been out for a week but servers were only down for a few hours if you want to go and moan at CCP go to WOW forums and join there club
Gaven Blands	Posted - 2007.05.25 11:17:00 - [136] The lag was perfectly timed. It got me a podkill that I shouldn't have been entitled to. And a free Vexor he ejcted from. If I could show you all the relevant GM tripe, you'd see why this podkill was so poingnant, what with it belonging to a macromass of paid accounts that can't be found guilty of exploiting under circumstances that singular paid accounts can be. I expect to found guilty of causing that lag last night and banned quickly for commiting the heinous crime of harrassing the macromass. Eve: Cheats prosper.
Fryke Caldari CRESCENT Inc.	Posted - 2007.05.25 11:20:00 - [137] As a programmer I know how you feel and I'm actually pretty happy you fixed it up as soon as you did. P.S. Also I'm curious why you have to reboot everyday. Why not once a week? Or even better, once a month?
Vergil 577	Posted - 2007.05.25 11:21:00 - [138] Originally by: Oratu lol, nerds. Pre deployment testing ftw !!!! Doh! iam a nerd too :( First Oratu welcome to the club Second it's amazing how people programed all of this I'd get a heart-attack and die... and I'm not even old enough for that and the amount of time they fixed the problem in is also good. only bad thing is I'm in the wrong time zone +10GMT makes DT fall on 9pm... but otherwise this is impressive running a whole universe with lots of solar systems. People will always complain there is nothing you can do about it, but you can't let it get to you. Live on EVE-Online.
Armoured C	Posted - 2007.05.25 11:30:00 - [139] wow you lost you pod big deal it only a game, just get on with you eve life and work your way back up, do you think they would of done this intentionally or something to personally spite your pod to death because they hate you that much? i am no programme i am a aeronauticle engineer so i dont know what half of you are on about but i am sure they did what they did because it was necessary to keep the server going, i lost a ship in the lag to but i rather they closed the server to stop the lag rather than keep it open and sit in a station doing nothing while i cant even type a message in the type box, and i also rather they do the matainince a hour a day than let say 1 day a week or 2 days a month or what have you least your only a hour away from game time instead of waiting a intire day but even if CCP did that some of you would still probably moan anyway
Stephar The High Priest	Posted - 2007.05.25 11:31:00 - [140] Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's.

Armoured C	Posted - 2007.05.25 11:37:00 - [141] Originally by: Stephar Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's. lol i loved to see them work there for a day and then see how much they get slapped and proven wrong ...
The Power HighTech Marines Ltd. FREGE Alliance	Posted - 2007.05.25 11:53:00 - [142] Edited by: The Power on 25/05/2007 11:59:19 Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. Oh how I like this ... We have alot of ppl running around blaming CCP for not having the servers up. And now they start the religion war between Unix and MS. Sure MS can run large servers .... http://h20341.www2.hp.com/integrity/cache/342431-0-0-0-121.html I'm DBA on an MS sql server set-up of 3 x 16 cores each with 32 gb ram. I am not saying MS is the only way. hell no .. but sometimes a company has to choice the platform they'll run on based on the kind of ppl thay can hirer I dont want to go into details about CCP job and how they are doing it.
Neyland	Posted - 2007.05.25 11:55:00 - [143] Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. Windows 2003 R2 Datacenter 512 MB of RAM required; 1 GB of RAM recommended; 128 GB maximum for x86-based computers; 2 TB maximum for x64 and ia64 computers Maximums: 64-way capable multiprocessor machine supported for 64-bit processors and 32-way capable multiprocessor machine supported for 32-bit processors http://technet.microsoft.com/en-us/windowsserver/bb430827.aspx
Gaven Blands	Posted - 2007.05.25 12:02:00 - [144] Originally by: Armoured C omgwtfbbgreadingcomprehensionftwlolbbqoneoneelevenbbqnerfthedrake In the words of the late great Bill Hicks.... "Well........... looks like we have ourselves a reader" Later, while I was hanging from the tree..... Eve: Cheats prosper.
Dark Flare Caldari Corpus PCG The State	Posted - 2007.05.25 12:06:00 - [145] Originally by: Nezz Jaran Edited by: Nezz Jaran on 25/05/2007 09:00:20 Cool, so the server farm is "now fixed" and it's time for me to go to bed (1 am). Glad I had a chance to ******* play tonight. Don't like playing when the servers are bouncing around, too much of a chance for something to go wrong. sigh Think of how LITTLE unscheduled downtime there is, and how much work goes into this game from CCP, then re-evaluate your little whine about missing one nights play. Good job CCP, you're doing some excellento work! I manage servers at work, and I sure as hell wouldn't want to have to manage as much, and as complicated as you do. Keep it up Dark Flare - Corpus PCG
Defiant952	Posted - 2007.05.25 12:10:00 - [146] You know, All of you complaining about the server stability and crap, How often are there problems like this? I mean come on very rarely does this happen and as to the "four years old it shouldn't happen" How many games do you play that support 30k+ simultaneous players AND the new content and features EVE puts out? You should be thanking them for the rare problems like this they have all things considered. Anyways, Keep up the awesome job CCP don't let the ones that do nothing but whine bother you.
Cryten Jones Gallente Infinite Development Consortium	Posted - 2007.05.25 12:11:00 - [147] Originally by: Tiger Ma SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. The EvE cluster used solid state disks already mate. Please see the following links and quotes. http://myeve.eve-online.com/devblog.asp?a=blog&bid=303 http://www.superssd.com/products/ramsan-400/ Quote: The new Tranquility consists of 70 x IBM LS20 AMD Opteron Blades, each Blade sporting two 2.4 GHz 64-bit Opteron processors with 2-4 GB of RAM, depending on application. That's a 140 CPUs, a full replacement of the current 140 Intel XEON 2.8 GHz CPUs. Hope that helps. -CJ
Saturous Viper Squad Triumvirate.	Posted - 2007.05.25 12:13:00 - [148] Horrah, the servers up'ish. Stop moaning and get your moneys worth whilst you still can. It makes me laugh how people ***** all night about it and once its up they still go on about it and waste more of their time. Yeah its hard to run a stable server for 30k-ish players at a time. My only guess is anyone who complains about 'lag' and 'disconnecting' has never played other MMORPGS and such. They all do it. "The key to success is the ability to put up with failure." - Follow that and everything will be fine.
Lowa Gallente North Star Networks Cruel Intentions	Posted - 2007.05.25 12:16:00 - [149] Wooohoo! Nerd discussion! I'm SO in! First, to all the people yelling about SQL vs Oracle etc. Get clue or shut it. <3 [Nerd mode] Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. True. But if the application does not scale over all cores then you can have a 100 and it wont matter. (Not saying it doesnt but hardware as you know only account for so much) And as you said, CCP aint going to change their investment. But hell it would be cool to see what the difference would from be a pure tech-geek possition. Originally by: Tiger Ma (Although think about solid state devices for all your tempdbs; it gets you past the 16Gb memory limit without architectural changes.) Just as an FYI, they have been running Texas Memory systems SSD's for quite some time. And recently upgraded them from 64GB to 128GB each if I remember correctly. Each of those can do like a gazillion IO's with "pretty good" response times. I believe bookmark-tables (before removal), market, inventories and some other things are placed on those beasts. Is that the salvation of all? No, but it sure does help. Originally by: Tiger Ma SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. No. Plain and simple no. In comparision with the high-end storage systems a single disk or DAS will look like a squashed snail. And in comparision with the SSD systems...Just mentioning the IOs handled by the SSD would make any physical disk explode. What you need to think of is a couple of things. 1. This is an extreeeeeeme multi-threaded environment (Obvious, I know) and the word "random access" takes on a whole other meaning A normal disk, even a high-end storage system, can not come close to handle the requests EVE generates (Damn players!). If you take a look at www.storageperformance.org you'll see that even the fastets systems would a) not stand a chance or b) cost an enourmous amount of money (And still be beaten by the SSD system who also are tested) 2. A single disk can look good in a single threaded test with low IO. Take into consideration that the fastets disks today can generate about 300 IO's (Vendor saturation tests can show close to 400) and in a storage system the fastest will give you about 250 IO's / disk. The reason for seing high numbers on DAS or single physical disks sometimes normally comes from two things: 1. IO's are cached in the server 2. IO's are cached on the physical disk (a HUGE risk of corruption comes into play there and its a big NO NO to use!) Now, that means as long as you short-stroke the disk, only run like 2k (or even 512b) IO's you will see 300-400 IO's and very few real applications generate those kind of IO's. Hell, both Oracle and MS SQL shoots out 64k or even 128k IO's and that is either by DB design or forced by the file system. You could go RAW on Oracle for example but Oracle them selves hardly promotes that anymore. I dont pretend to fully understand the EVE environment (and trust me I've tried) but I have a pretty good idea. I do know that its so many different things going on that there is no single thing that will fix all problems. [/Nerd mode] Sorry about the rant, its not directed to you Tiger specifically, I just saw my chance to talk nerd stuff about something I actually have a black belt in. :) Cheers, Lowa What if the truth was something else?
nomis ekoobmah	Posted - 2007.05.25 12:24:00 - [150] Originally by: Stephar Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's. Does a 4096 node Linux cluster count? I used to write massively parallel applications for an oil exploration company.

Angelis666 Amarr The Knights Of Camelot DeStInY.	Posted - 2007.05.25 12:26:00 - [151] I Would kill ANYONE to have the nerdy knowledge of some of the people in this thread!
Oghma Grianainech Minmatar MyPartyBase Fimbulwinter	Posted - 2007.05.25 12:43:00 - [152] Originally by: Devious How many games have you played that are 3-4 years past beta where the dev's and support staff tell you what went wrong and what was fixed or will be fixed??? i can count only one. EVE Online. Tell ? :) Come on now, any bussiness is the same, they only tell you what seems to worth telling and keep the rest a secret. I think you should try some Real life business PR work. :p See how much truth there is in any PR work.
Oghma Grianainech Minmatar MyPartyBase Fimbulwinter	Posted - 2007.05.25 12:45:00 - [153] Originally by: Neyland Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. Windows 2003 R2 Datacenter 512 MB of RAM required; 1 GB of RAM recommended; 128 GB maximum for x86-based computers; 2 TB maximum for x64 and ia64 computers Maximums: 64-way capable multiprocessor machine supported for 64-bit processors and 32-way capable multiprocessor machine supported for 32-bit processors http://technet.microsoft.com/en-us/windowsserver/bb430827.aspx Sorry I have to LOL when someone quotes something from MS website, have you seen the video interviews made by slashdot unixbsd and others at MS so called www HQ anmely m$.com? :P I suggest you check those out then quote form their web.
ry ry StateCorp	Posted - 2007.05.25 12:56:00 - [154] i wanted the performance figures for a ford, so i contacted general motors via www.wehateford.com and they were able to inform me that 'fords suck'.
Twiekdon FinFleet	Posted - 2007.05.25 13:07:00 - [155] **** happens, deal with it people.
Cryten Jones Gallente Infinite Development Consortium	Posted - 2007.05.25 13:17:00 - [156] Originally by: Twiekdon **** happens, deal with it people. My GOD dude, your sig is sOooo going to be nerfed :D -CJ
Borg 5105	Posted - 2007.05.25 13:27:00 - [157] I hear it so obviously often : MicroSoft sucks. to keep it really short : Yes, MicroSoft has/had MAJOR flaws in their software... but then again which software developer provides bugfree Software anyway? Linux Opensource is ONE BIG BUG as well. Which brings me to the consecutive statement that Linux, unlike MicroSoft's operating system is not really maintainable by the average computer user. It takes more than average computer skills to use Linux properly. Result is, more people use MicroSoft Windblows and I am quite sure MS SQL is much more compatible with MicroSoft Operating Systems than with Linux, Unix or whatever more. Yours Sincerely, Borg 5105... or : How do I install that TAR FILE!! RESISTENCE IS FUTILE ;-)
Nymos Celtic Anarchy Anarchy Empire	Posted - 2007.05.25 13:29:00 - [158] oh really, eve has database problems? that's... so 2006... database fubar, lag, bugs, borderline exploits, but yea we can walk in stations soon and we'll get a whole load of new bugs, err features, with stuff like "heat". this morning teaches me one thing: never ever log on in the morning and switch to a short skill while getting ready to leave for work. i almost missed the train because i don't want to miss hours of training :P -- Every time a carebear dies an angel gets their wings (murder one)
DEVILSENIGMA Roving Guns Inc. RAZOR Alliance	Posted - 2007.05.25 13:55:00 - [159] Originally by: Xrious Edited by: Xrious on 25/05/2007 10:54:06 I'm sorry, I'm not aiming to CCP bash, but the stability of the system isn't good enough IMHO. SQL server's don't need to be rebooted everyday, they stay up for years, not 23 hours. As someone else pointed out, running the forums and the website on the same cluster as the game? great, game server goes down, the players can't even find out about it. Someone said that how often does a company give explanation's on why a system screwed up 3-4 years out of beta, no offense but a system shouldn't be having these kinds of problems 3-4 years out of beta. Giving explanation's on what has happened is great, we really do appreciate it, but I think the player base demands it. Eve is an intelligent, strategic game, not a reflex FPS and the type of player tends to be more mature (well, mostly ) that you'd find on other games out there. A vast percentage of the player base understands the technology behind Eve so they could hardly try to cover it up. Edit: typo Here is the flaw in your argument - During Beta the servers at max had a few 100 people online, such problems usually occur when thousands of people are online - like last night. The server and database itself would have changed since beta due to new features and improvements The problem occurred due to a much newer feature - Contracts, didn't exist Logic which applies to regular software and web-services doesn't apply to a game like Eve. Eve's codebase is evolving, and the players are pushing Eve to the limits. Such things are bound to happen. --- Eve Database \| TQ Server Status Coldfront Come in and feel the chill
threetwoone	Posted - 2007.05.25 13:58:00 - [160] Originally by: Borg 5105 Which brings me to the consecutive statement that Linux, unlike MicroSoft's operating system is not really maintainable by the average computer user. It takes more than average computer skills to use Linux properly. CCP staff have just average computing skills ... right? O_o

Bambi	Posted - 2007.05.25 14:15:00 - [161] I think to understand why CCP use M$ products you have to look back to when EVE started taking shape, several years before beta testing was even on the table. The state of Linux was not as well organised as it is these days, and M$ was really a sensible option. 10 years later things have changed, and M$ may not be the most valid choice, but its what we have. I know I wouldnt want to take the EVE cluster and try to port it to a nice new unix cluster. As always we get the whingers who bemoan a loss of a days game time, GET A 'KIN LIFE!! It may be THE BEST game in the world but it is still only a game. (Can't belive I typed that) Keep up the good work in CCP-Land!
anubis deathbringer Caldari Fluffy Rabbit Killers	Posted - 2007.05.25 14:16:00 - [162] Edited by: anubis deathbringer on 25/05/2007 14:15:45 first off for all the complainers, shut up. if you complain in hopes of gtting discount or what ever, forget it. if they were going to give discount then they would. secondly M$ might suck but if anyone here wants to change to oracle then by all means offer your services for free. cause the differences arent worth the money they would spend changing over and well we would all be enjoying the time without the game while they did the cross over. no offence to oracle, however when your first starting up a game like this. poeple tend to go with cheap and fast, instead of paying 10000+$ for oracle licence when they started up the probably got a better deal with M$. well its all about business, still the lag and the requirements and the downtime isnt as bad as other games so i prefer this one. as for the attack on the poor junior admin. if he didnt have the power to reboot the game and fix issues then when issues come up we would have to wait for the admin to come from home and deal with the issues and then reboot the system or at least wait till he wolk up and remotely rebooted. which could mean hours. also i doubt that the admin would want to pay someone just to sit there and watch the server with no power to do anything. which is a complete waste of money if they have no access. besides all this. like they said he is a junior admin, we have all been there and had to learn the hard way. at least this one was smart enought to realise that after the 1st reboot they couldnt fix the issue. so they informed the right people and got it fixed. otherwise it might have been rebooting till morning and then when the admin came in with thier coffee they would have started to fix it, meaning we might have been down for a full day instead of a few hours. big clap to the junior admin who saved us. that being said i could enjoy a small make us happy payment of maybe something like 1mil isk or 1 point in the skill of our choice or something like that could help. ;) P.S i love this game! and you guys and gals over there watching our game Rock! pps. can someone fix my picture for the thread? :D thanks
Borasatar	Posted - 2007.05.25 14:19:00 - [163] Originally by: Thodoros SQL server??? Now you cant be serious about it? If thats true now we know why we have so much lag all the time. No wonder the server cant handle big fights with over 100 peers on the same system. Very sad. Actually, the problems are more likely due to other reasons than this. There seem to be a number of bottlenecks inherent in the design of both the client and the server side code.
Jai Centarium Anqara Expeditions The OSS	Posted - 2007.05.25 14:20:00 - [164] I'm going to posit this tidbit: How many players would cancel their subscriptions if the servers continue to bounce, a small fleet action causes HORRIFIC lag... this far into a mature game and 30v30 brings the system to a halt?... and training time (one of the GREATEST selling points for EVE, IMO) continues to be lost because you tried to view a contract in Jita, and have been locked out of the game ever since? Now, how many players would cancel their subscriptions if CCP announced on Monday that all the features of Rev 1.4.3++ are on the shelf indefinitely. No "heat." No POS 2.0. No constellation sovereignty. No corp interface improvements. No new features. Just say no. Personally, I'd gnash my teeth at the loss of some of the new features, but I'm not losing any sleep over it. I'd much rather have a rock solid online interface, capable of delivering those incredible-scale fleet battles at something greater than a slideshow. The blob isn't just a cause of lag, but a result of it... I'd rather be in a massive blob with the lagspikes start hitting than in a rover group trying to pull some time-sensitive manuver. A solid engine will lead to memorable, fleet-scale fights. And memorable experiences, not flashy features, drive subscription numbers. Once upon a time, we were promised a "Need for Speed" initiative. Deliver on that promise, then work on adding new shiny things to the game. A draconic approach, to be sure. But sometimes, every problem IS a nail. I want my disembodied head avatar back. Jai's Guide to Research
Lesch Gallente Meyvn Corp	Posted - 2007.05.25 14:24:00 - [165] Yay for junior db admins in charge during North American prime time. I think we may have found the weakness in your support strategy.
Riddick Valer	Posted - 2007.05.25 14:28:00 - [166] Of course a Junior admin is in charge. Its late at night/early morning local time for them. I'd rather have a junior admin shut down the server and possibly fix it or prevent further errors then wait for them to pry the senior guys out of whatever bar/bed they are in.
ry ry StateCorp	Posted - 2007.05.25 14:31:00 - [167] and he's assuming junior = not as good, and you know what they say about assumption...
Nina Mires Sharks With Frickin' Laser Beams Mercenary Coalition	Posted - 2007.05.25 14:41:00 - [168] Originally by: Pallidum Treponema Now, if I'm not entirely mistaken, "SQL Server" in this case actually refers to a fairly large cluster with several individual high-end servers. All of those machines are multi-processor and possibly multi-core as well, with maxed memory (16GB or more each), attached to a SAN (Storage Area Network). Still, it gets presented as a "server", which makes people think of the one single machine they're used to. Unfortunately although Clustered SQL Server will do Active / Active, i.e. more than 1 machine giving out SQL Queries, there is only ever 1 computer doing this, for 1 instance. Yes it can be more than this but it requires transaction log mirroring and such which would add to SQL delay, and possible corruption of the data. The only way to get around this is to run multiple seperate databases on seperate SQL instances which would be running on a different node in the SQL cluster. But this would involve splitting the existing database design / structure into things that could be split off . Such as the forums to some extent, and possibly contracts + who knows what else.. only the DBA / developers i suspect. Anyway personally keep up the good work CCP, oh and look at replacing those HS20's soonish ;) /me points in the direction of HP C Class. Nina ---------
Ammath The 5 Amigo's LLC. NxT LeveL	Posted - 2007.05.25 14:50:00 - [169] You know in many other MMOs when this stuff happens they do a "double XP" day to make up for it... Just a suggestion CCP if you want to quiet 90% of the hatemail down and be able to take the servers down until you really fix it instead of the up/down games just adopt this policy... tell people that the next when the servers finally come up you will increment the skill-timeline forward 24hrs for every active training skill... Just a thought.. Ammath Director The 5 Amigos LLC
Miko Li	Posted - 2007.05.25 14:52:00 - [170] Originally by: Jai Centarium I'm going to posit this tidbit: How many players would cancel their subscriptions if the servers continue to bounce, a small fleet action causes HORRIFIC lag... this far into a mature game and 30v30 brings the system to a halt?... and training time (one of the GREATEST selling points for EVE, IMO) continues to be lost because you tried to view a contract in Jita, and have been locked out of the game ever since? Now, how many players would cancel their subscriptions if CCP announced on Monday that all the features of Rev 1.4.3++ are on the shelf indefinitely. No "heat." No POS 2.0. No constellation sovereignty. No corp interface improvements. No new features. Just say no. Personally, I'd gnash my teeth at the loss of some of the new features, but I'm not losing any sleep over it. I'd much rather have a rock solid online interface, capable of delivering those incredible-scale fleet battles at something greater than a slideshow. The blob isn't just a cause of lag, but a result of it... I'd rather be in a massive blob with the lagspikes start hitting than in a rover group trying to pull some time-sensitive manuver. A solid engine will lead to memorable, fleet-scale fights. And memorable experiences, not flashy features, drive subscription numbers. Once upon a time, we were promised a "Need for Speed" initiative. Deliver on that promise, then work on adding new shiny things to the game. A draconic approach, to be sure. But sometimes, every problem IS a nail. /signed

svarog tupan	Posted - 2007.05.25 14:52:00 - [171] Just by curiosity, could u post the query of doom ? Sounds liek you used a bad query that generates a biiig cartesian product.
Brigitte	Posted - 2007.05.25 15:02:00 - [172] ffs its ok if you didn't wana log in to set new skills but for us tht couldn't its tough yet again soddin server fix the lagg ffs and all the other stuf still broken let the voice crap alone
RcCypher	Posted - 2007.05.25 15:03:00 - [173] All I would like to say to you guys at CCP is, S*IT happens, thanks for pulling the long hours and getting it running again. And huge props to anyone who got called back in to work in the middle of the night. You guys rock! RcCypher
DEVILSENIGMA Roving Guns Inc. RAZOR Alliance	Posted - 2007.05.25 15:04:00 - [174] Originally by: Lesch Yay for junior db admins in charge during North American prime time. I think we may have found the weakness in your support strategy. You are assuming that there are more North American players than European players. However, there are more European and Non-North American players playing Eve than North/South American players. I beleive this was mentioned by Oveur or kieron somewhere as well. --- Eve Database \| TQ Server Status Coldfront Come in and feel the chill
Radgun Minmatar Republic Military School	Posted - 2007.05.25 15:13:00 - [175] well, can you keep it from happenning again, please ?
Seiang Tsukiko	Posted - 2007.05.25 15:34:00 - [176] Originally by: svarog tupan Just by curiosity, could u post the query of doom ? Sounds liek you used a bad query that generates a biiig cartesian product. SELECT * FROM contracts
Luna Starr	Posted - 2007.05.25 15:40:00 - [177] I agree with many of the people on this forum regarding CCP responsing as fast as they could to problems as they occured. I can't believe people are betching about being down for 8 hours out of an entire month. For what we pay to play you could only see 2 movies in the entire month without popcorn and drinks. Thank You CCP for how quickly did you did fix the problem and keep up the great work on the high wire act over a fire pit with cross winds blowing. :)
Audri Fisher Caldari VentureCorp Imperial Republic Of the North	Posted - 2007.05.25 15:41:00 - [178] Originally by: Xrious Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? I apologise and know that you need sleep, but employ someone else. Thats really what we're paying for. We all abuse M$, but I had alot less problems with SP2 than CCP seems have with their own systems. Sorry, just had the message pop up on my screen telling me that the server is shutting down at 11am for its normal reboot. Very very anoying when you've spent hours getting in. Kinda rubbing salt in the wounds there CCP. X Sig? what sig? if they fire the junior dba, that means they will have to hire someone with even less experience with there system. I do not know how CCP runs it's shop, but the most successful organizations operate on the principle that decisions are made at the lowest level possible. This has several major benefits, among them extremely fast decision making and turnaround time. A great side affect is this also keeps the higher tier guys from being burned out trying to micro manage everything, while it trains the junior guys to handle there bosses job. It also helps to eliminate pointless fiefdoms and red tape. Once you deal with low level government employees that can't pick there nose without 15 forms filled out and approval from 3 levels of bosses you will truly appreciate this.
the plague	Posted - 2007.05.25 15:42:00 - [179] after all this the people i feel sorry for is the peeps tht set there alarm clock 10mins before they actually have to get up for work just so they can change a skill! my question is how many peeps actually lost valued sp today because of this? quite a few i would say! maybe money is an issue for ccp, maybe they dont earn any money for soft and harware maybe they blow it all on corporate jibber jabber either way alot of ppl pay there hard earned cash to play this game and are being stopped because you have weeners running the servers! sounds like a few peeps who have posted allready know what there talking bout, maybe ccp should think bout employing some of them oh one last this...... whats with all the LAG even after 4 years you cant fix it bahhhhhhhhhhhhhhhhhhhhhhhh
winglessangelxxx	Posted - 2007.05.25 15:54:00 - [180] Originally by: CCP Valar A quick post mortem after this horrible night. The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables. The SQL Server is now back to its normal self. The development team will hopefully have a proper fix for this ready soon. And now back to bed Explain how a patch to fix voice services screwed with the contracts system please.... I'd love to hear this....

Hebik Fane Havoc Inc	Posted - 2007.05.25 16:02:00 - [181] Everyone stop it! I just finished the 3 day MS 2780 SQL class and my head is going to explode here. Deadlocks, tail of the log, stored proceedures I must now pod you all. P.S. For those of you that don't know, there is a reason there is a 3 day class on just maintaining your SQL server, its not easy so cut the Devs some slack.
Rude Bwoy	Posted - 2007.05.25 16:16:00 - [182] why don't they buy DB2 or Oracle. i.e. a REAL db. Who's the rudest of them all!
Astrohole	Posted - 2007.05.25 16:18:00 - [183] Ah, just make me a standalone version of Eve and then I can gank myself if it stumbles.
Defiant952	Posted - 2007.05.25 16:19:00 - [184] Originally by: winglessangelxxx Originally by: CCP Valar A quick post mortem after this horrible night. The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables. The SQL Server is now back to its normal self. The development team will hopefully have a proper fix for this ready soon. And now back to bed Explain how a patch to fix voice services screwed with the contracts system please.... I'd love to hear this.... They didnt say the patch caused it, What caused it was a bad query on the contract system based on the global region settings that they were never designed for. Cut them some slack and once again as others have said, This is not the same EVE as 4 years ago so stop that argument already. You all should be praising them for how few unshceduled downtimes you have.
Korotani Caldari Love and Rockets	Posted - 2007.05.25 16:28:00 - [185] Thanks to whoever stayed up all the night long to fix this - People may whine about it, but I doubt they could do it, and to all you computing A levelers or degree students out there, you think you can fix it? Stop being a self righteous c*** and apply for a job at CCP, then we can take it out on you when you get it wrong.
lam0r Caldari The Legion.	Posted - 2007.05.25 16:31:00 - [186] Originally by: Xrious Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? I apologise and know that you need sleep, but employ someone else. Thats really what we're paying for. We all abuse M$, but I had alot less problems with SP2 than CCP seems have with their own systems. When I was working in DBA I often had the same power to kick 10-15k people off our system in order to prevent 'collateral damage'. When you have scripts updating numerics, it's hard to 'undo' your mistakes unless you have an audit/transaction log (which would take even more downtime to check and restore data). As far as MSSQL goes, well it's a fast and robust DBE that is so easy to administer. I found it easier than my time administering Oracle servers and as many have said, the difference in performance is really negligible at that scale (I've had workstations running MSSQL kick the crap out of any other windows based DBE, so imagine what multi-CPU machines can do). My curiousity lies in the specs of the DB server and how it is configured (multi-threading? using stored procedures? is it performing unnecessary execution plans?) but that's what our friendly Dev team is for I guess :P
Jim McGregor	Posted - 2007.05.25 16:32:00 - [187] Edited by: Jim McGregor on 25/05/2007 16:32:09 I tried to log on this morning from home to change skills but found that the server wouldnt accept logins. So when i got to work I downloaded the Eve client to my work computer and was on my way to quickly log in and change skills when my boss showed up in the exact same moment as Eve started showing the fullscreen intro scene... Lets just say he wasnt happy. --- Eve Wiki \| Eve Tribune
Pang Grohl Gallente	Posted - 2007.05.25 16:37:00 - [188] Originally by: CCP Valar A quick post mortem after this horrible night. The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables. The SQL Server is now back to its normal self. The development team will hopefully have a proper fix for this ready soon. And now back to bed I knew allowing global viewing of Contracts was a bad idea. Anyway, good job on getting things sorted. Chin up to the Junior DBA, his reboot let me get in to set things up for an extra long weekend away from EVE. Si non adjuvas, noces (If you're not helping, you're hurting)
Defiant952	Posted - 2007.05.25 16:57:00 - [189] Edited by: Defiant952 on 25/05/2007 16:55:51 I swear half of you don't read. Look if this was the same as it was FOUR years ago it would be fixed, BUT how new is the contract system? Thats right pretty new especially for the global viewing which was what caused the random problem. EDIT: Woot top of page 8 :P
Dikat	Posted - 2007.05.25 16:58:00 - [190] Originally by: Angelis666 I Would kill ANYONE to have the nerdy knowledge of some of the people in this thread! And I'm sure THEY would kill to know the touch of a woman :)

Milz0r	Posted - 2007.05.25 17:00:00 - [191] Yes, i understand that. But if it took them 4 years to identify a contract problem with viewing all regions...then there is something wrong. It just seems that they dont really fix anything...they just restart the servers. I lost a 66 million isk blueprint and a raven to this recent server crash. I sent a petition and i bet they wont do anything about it...even if i am a paying customer to a game that is constantly crashing.
torswin Caldari Capital Productions Inc.	Posted - 2007.05.25 17:02:00 - [192] I wonder why the Junior Database-guy get so much flames? He did what he thought was right and when he discovered that his "fix" didn't help, he contacted someone who probably had more experience than he did in analysing the cluster (hope i got that right, please correct ) and it is inhuman to be perfect
Jim McGregor	Posted - 2007.05.25 17:04:00 - [193] Originally by: Capt Tripps Hate to tell ya but in my IT experience starting in 1982, it never ends with computers and servers and software, always another problem. That's just how it is. Thats why we get paid the big bucks to fix it. --- Eve Wiki \| Eve Tribune
UmnaHun	Posted - 2007.05.25 17:06:00 - [194] Edited by: UmnaHun on 25/05/2007 17:06:59 Soo many clever people here... Why I only see one - and only one - eve like game running??? If you are so clever/experienced/talented, do a better one! lol! CCP Keep up the good work (and nerf BOB)
Megasexmanaut	Posted - 2007.05.25 17:10:00 - [195] Since Eve is a constantly evolving project, I expect it to have occational health problems. That is not really a bad thing because it's always getting better. What I like about the devs at CPP is their candor in letting me know what the problems are about and what they are doing to fix it. Having a better understanding of the situation removes my misconceptions and elevates my respect and appreciation for the people at CPP.
lam0r Caldari The Legion.	Posted - 2007.05.25 17:13:00 - [196] I would say just keep it clean and the feedback constructive, without us it's sometimes hard to know to what extent a problem exists! Sometimes very important points are missed through the sea of flames :P
Garth Vaders	Posted - 2007.05.25 17:15:00 - [197] Hello to all. This is my first post in these forums. I decided to try out this game recently without much hope for it but got immediately hooked. So I upgraded my subscription to 6 months :) The concept is great and i like almost everything about the game (although i have problems with the blueprints manufacturing and other similar "heavy" stuff like that but i have great fun so far doing all the rest.I would appreciate any links on how the research and manufacturing fields are working cause i find the subjects a bit confusing. Now to the point. Although the game is great and i have played many online games i see that it has severe lag problems. It shouldn't be so laggy with only 30-40 thousant people on. It is sad to say that my greatest fear when i click on a mission is if i will make it back alive not because i am afraid of the dangers of the mission ahead of me but because i am afraid of lag. The other day (not this one that we had the "problems" this thread is refering) the server got a lag spike at the exact time when i was warping desperate into another gate in order to avoid a swarm of pirates that had droped down my shields and half my amror. So i had lag spike and then got disconnected. And well i went mad cause i had suceeded the mission and the only thing i needed was to be able to warp safe back and .... lag was making it impossible. Anyway after several minutes of agony i managed to log back in and found my ship to the safe system i had warped .Thankfully the jump was made even if i wasn't there. Seems the server managed to execute the warp command just before it boots me.... :) I say all that to share with you the horror that us gamers get when such things happen but i bet you are all familiar with it. As far as last night goes my 2 cents are: 1) I really appreciate the friendly way the admins communicated us the problem they have :) 2) The fact that you are running a Microsoft server perlexes me. I was almost certain that you would run the game in a Linux or Unix server because i was thinking perhaps in an oversimplified way: "Hey these guys are Scandinavians the must use Linux" :P So my suggestion on this matter is try to switch to Unix or Linux if you can it will save a lot of lag. Get some Linux expert (you are Scaninavians i am sure you find many Linux experts up there north hehe) 3) Someone mentione that these forums share the same resources with the server hosting the game so it's common sence to suggest to remove the Website from the server and set it up on a server differnt to the one runing the game. 4) To remediate on the lag that a universe so big hosted in one server has i would suggest the following solution: " Divide the universe in 2 servers.So server A will have half the universe and server B the other half. Make also a server C to contain all the "transactions info" in effect set the whole "market" thing on that different server C and have it communicate the "market" data with the other two. The universe will still remain the same ONE it is . You can convert certain jump gates into transitions from server A to server B so when a ship jumps will get transitioned to the other server. This way lag will be diminished much i think. 5) You have to understand that since this game is GREAT it will keep on growing BUT if these additional new players (like me) bring lag with them then the whole world will suffer from lag making actually other players leave. (Imagine it as if trying to hold sand in your arms, at some point the more sand you will grab the more will slip away)which is counterproductive of course) So what will happen is that you will gain players everyday while losing others at the same time that won't be able to put up with the lag the new players will bring. :( So if you won't find a way to remediate the lag the player numbers in all will stop increasing >meaning less profit for your company. Bottom line: Fix the lag before is too late
Lowa Gallente North Star Networks Cruel Intentions	Posted - 2007.05.25 17:17:00 - [198] Nina Mires, my dear, the HAVE split the DB's up. The SSD systems are a prime example running one single function (or DB table). The rest is split across hardware too afaik, it may look like a single instance but it isnt. How that is tied together I'm still figuring out but understanding the environment takes time. Originally by: Dikat Originally by: Angelis666 I Would kill ANYONE to have the nerdy knowledge of some of the people in this thread! And I'm sure THEY would kill to know the touch of a woman :) That reply deserves the "touche!" remark tbh! But, in case you didnt know, nerds are the new black. And we are in hot demand! Honest to [who ever you believe in] its like a revelation when the ladies realize that finger/hand dexterity (sp) is a HUGE "I Win Button"!. However, taking a shower now and then helps of course. /Lowa <-- Showered and ready. I'm just gonna log in for a while... What if the truth was something else?
Ulitio	Posted - 2007.05.25 17:24:00 - [199] This is not to be meant as criticism. This whole episode makes me wonder if the current software engineering practice to use off the shelf products for purposes really works. What I mean is MS SQL server(and most RDBMS products) are designed as business products. They are not really designed to be used in any real time applications(and while MMOGs aren't RT apps, they are closer to RT than a pure business model.) Now I know what the vendor's say, but the fact remains is how the product was designed, not what some marketing bots say it can do. So I have to wonder if the storage mechanism was specially designed, if this sort of performance problem would have occurred(And yes I am well aware of the implications, I have spent the past 4 years or so trying to adapt OtS products to specific purposes with good success. However my experience is that when you bring someone new in, even if they have experience with the OtS product, they still get confused easily by the mods to it) Food for thought...
Siege Minmatar Siegecraft Bounty Hunting	Posted - 2007.05.25 17:29:00 - [200] Originally by: Milz0r Yes, i understand that. But if it took them 4 years to identify a contract problem with viewing all regions...then there is something wrong. Errrr.... Contracts themselves have only been around for about 6 months now, and globally viewing them has been around even less time than that. Maybe, what, 2 months? So, explain to me again how they could have found the problem 4 years ago, on a two month old system.

Hondo Kimotoro Farmer Killers United Corporations Against Macros	Posted - 2007.05.25 17:51:00 - [201] To those who were blaming the jr techie guy, and ccp for not having a high end guy on duty at the time. This isnt unusual, in fact the military ( US ) does this as a common practice, you think the general is going to be up guarding the motor pool or pulling cq duty? heck no, some pvt or sgt is gonna be doing it. Then when something beyond their abilitys to cope with occurs they go up the chain of command. and thats exactly what happened here, guy did what he could, was beyond him, he woke the general and he stormed in to the rescue :-) that being said, the system has grown more instable since i first started pre bloodlines. But ive read some initiatives they are trying to work on to fix this. And i congradulate them for at least trying to fix it. To those of us who are upset the server is down,( yes i get ****y too when it happens.. but its anger out of love :-P ) i simply point to the other games out there, my fleet battle account might just take a vacation with all the lag i get fron fleet operations until it is fixed and go play another game ( my other accounts however will stay nice and active.. so NO you cant have my stuff im coiming back :-P ) then when the server goes down here i can just swap to a different game, best of both worlds. however getting rid of some of those macro mission runners and miners might help the lag a little bit <.< >.>
Aterna Talon's Grasp	Posted - 2007.05.25 17:53:00 - [202] Originally by: Garth Vaders Hello to all. Stuff 1) We all love our devs. Some people do the "tough love" thing though. 2) Read some of the posts earlier on. If they don't turn your ears to goo, you will probably understand. If they do, then you don't know enough about servers and DB admin to make any qualified remarks (I don't). 3) CCP has already mentioned that the website server and forums are being upgraded, someone from the web cell is working on it. Check the recent dev blogs? 4) EVE already does this, but on a much larger scale then you know. The universe isn't divided into just 2 or 3 servers. It is divided into over a hundred 'nodes.' Each node supports a small group of star systems. Any time you use a jumpgate or change system, you move from one server to another. The DB servers are already not the same server that the system nodes are on. The market, the contracts, bookmarks, hangar, etc are all on their own server array. Again, read some of the dev blogs and above player comments, before you post. 5) You stopped making sense somewhere around here. If everyone would spread out and away from mission hub systems and market hub systems, it would lessen the lag felt on those nodes. EVE can support a lot of players, but the amount of people you can cram on a single node is still finite, and the more NPC's, wrecks, missions being run lower that number significantly. Simply put, EVE needs a mission agent rework, to spread them out a bit to little traveled systems, to ease node stress on places like Motsu and Saila. I don't even know if it is possible, but a major solution would be some sort of 'Omgwtfhax' server code that allowed multiple nodes to support a single system, with the ability to add or remove nodes dynamically, on the fly so to speak. So that we don't have to ask in advance to reinforce nodes for epic battles, they can happen seamlessly and afterwards, the node count for the battle can drop down to normal while the victor takes the spoils and the loser flies home or gets a new clone. - - - WTB new sig, evemail me please.
The Kinetic Caldari Sword Production	Posted - 2007.05.25 18:02:00 - [203] Originally by: CCP Admiral Chamrajnagar Server is now online, we had a database problem where the database server went too 100% cpu and remained there for 1 hour. I made the decision after consulting with the GM's to take the server down gracefully instead of allowing it to melt down. A job had caused an abnormal load and the server was trying to catch up but could not. Upon startup of the cluster again a machine decided to reboot itsself and cause the infinite startup loop. After correcting that issue we are now online. My apologies for the delay in updating you. I must say, it is nice now you guys keep us informed as to whats going on and what caused the problems etc -------- hi2u
Lesch Gallente Meyvn Corp	Posted - 2007.05.25 18:10:00 - [204] Quote: I wonder why the Junior Database-guy get so much flames? He did what he thought was right and when he discovered that his "fix" didn't help, he contacted someone who probably had more experience than he did in analysing the cluster I'm not sure anyone's upset with the junior guy himself. From my perspective at least, it's simply poor customer service to have put him and the customers in that position in the first place. Quote: and he's assuming junior = not as good, and you know what they say about assumption... I wasn't assuming anything. If you read the dev post it was laid out quite clearly there. I'll paste it here for your benefit: Quote: When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. After I got a call shortly after the second reboot I identified the cause and tried applying fixes live. The server was however gone too far to recover so a reboot was initiated at 7:10. The server went down at 7:36. While the server was down, I forced a query plan on the contracts lookup proc that works in all cases and manually updated statistics on the relevant tables. pretty clear that in this case, junior = not as good. Now, as far as this goes: Quote: You are assuming that there are more North American players than European players. However, there are more European and Non-North American players playing Eve than North/South American players. I beleive this was mentioned by Oveur or kieron somewhere as well. I made no such assumption. I simply stated that this was allowed to occur during North American prime time. I understand that it's in some people's nature to want to stand up for CCP in a situation like this. I also understand that there are people who just want to argue with someone on forums from time to time. What I'm suggesting is that people look at the real issues going on here and make themselves heard when they receive inadequate service, so that the people we pay for that service might take notice and make efforts to correct the situation.
Remorer	Posted - 2007.05.25 18:38:00 - [205] Originally by: Garth Vaders It shouldn't be so laggy with only 30-40 thousant people on. ONLY 30-40 thousand.. Isn't that a lot of players on the same server??
Mari Onette	Posted - 2007.05.25 18:38:00 - [206] Originally by: Garth Vaders It shouldn't be so laggy with only 30-40 thousant people on. There is no other game in the world that supports 30 thousand users in a single virtual world. WoW has more subscribers then eve, but in terms of the number of people on a single world at a time, EvE beats it hands down. A wow server with 30 thousand concurrent users would be WAY more laggy then eve (actually, it would probably be outright broken). Originally by: Garth Vaders To remediate on the lag that a universe so big hosted in one server has i would suggest the following solution: " Divide the universe in 2 servers.So server A will have half the universe and server B the other half. Make also a server C to contain all the "transactions info" in effect set the whole "market" thing on that different server C and have it communicate the "market" data with the other two. The universe will still remain the same ONE it is . You can convert certain jump gates into transitions from server A to server B so when a ship jumps will get transitioned to the other server. This way lag will be diminished much i think. Eve is not a single server. It is actually a cluster of some 150+ nodes (i am to lazy to look up the exact number), each running 1 or more star systems in eve. Market and contracts are both on seperate nodes, as well as the webserver for the fourms (although that is a recent improvement). So your suggestion about seperating eve into different servers has already been done, its part of the basic design of tranqulity. The limitation of the system is that you cannot put more then 1 node on a single star system. Combine this with the limitations of modern hardware (its more a bandwidth issue then pure CPU power) and you start encoutering lag once you get 300 some people in the same star system. Jita and Motsu are perfect examples of this problem. They both have dedicated nodes and there are still lag issues. The simplest way to reduce lag in eve is to move away from core systems like jita and motsu. There are a few thousand star systems in eve, many of them completely empty most of the time. Make a system like that your home base and you wont encounter nearly as much lag anymore. I hardly ever encounter lag since I moved away from crowded systems, and I still have easy access to equipment, minerals, and good agents.
Toramt	Posted - 2007.05.25 18:57:00 - [207] Edited by: Toramt on 25/05/2007 18:58:35 Knowing something about how Oracle works (SQLServer is probably equivalent), query plans can change over time especially if any sort of structural change was made to any of the tables in the query, or even if the number of rows crosses an invisible threshold. This can cause the performance of the query to change drastically if the database decides that a very 'bad' query plan is the best way to get the data now. DBA's can override this behavior by hard-coding the plan or 'faking' the statistics on the relevant tables, but this is not usually done as the database normally does a good job of picking the right query plans and responding to changing table structure / sizes. In this case, it is quite possible that something tangential to the Contracts tables changed (the Items or Players tables perhaps), but since they are referenced they impacted the query plan for Contracts. This kind of problem is the sort you would not always see in a dev/test system, and it is reasonable for it to show up after X years of the Contracts system existing, since the affected tables may not have been core to Contracts at all.
Rigsta Gallente Raddick Explorations Executive Outcomes	Posted - 2007.05.25 19:28:00 - [208] Originally by: Milz0r Yes, i understand that. But if it took them 4 years to identify a contract problem with viewing all regions...then there is something wrong. Holy paradox Batman! Contracts have been around for maybe 6 months, the problem has been around for about a day and it's taken 4 years to notice it! </sarcasm> Originally by: Jim McGregor I felt the disturbance... it was like a million voices suddenly stopped whining for a second. Unfortunantly it then continued.
WhitePhantom Gallente Edenists	Posted - 2007.05.25 19:50:00 - [209] Originally by: Ulitio This is not to be meant as criticism. This whole episode makes me wonder if the current software engineering practice to use off the shelf products for purposes really works. What I mean is MS SQL server(and most RDBMS products) are designed as business products. They are not really designed to be used in any real time applications(and while MMOGs aren't RT apps, they are closer to RT than a pure business model.) Now I know what the vendor's say, but the fact remains is how the product was designed, not what some marketing bots say it can do. So I have to wonder if the storage mechanism was specially designed, if this sort of performance problem would have occurred(And yes I am well aware of the implications, I have spent the past 4 years or so trying to adapt OtS products to specific purposes with good success. However my experience is that when you bring someone new in, even if they have experience with the OtS product, they still get confused easily by the mods to it) Food for thought... Eve is a business product, how does it not fix the exact purpose of MSSQL?
Jimer Lins Gallente Sanctuary	Posted - 2007.05.25 20:02:00 - [210] Just remember to take the scheduled job that runs UPDATE STATISTICS every hour out when they do deploy a better query. I've had to do something very similar in my job; we had a query that worked perfectly in test but took forever in a production mode. It turned out to be related to how the query plan was being optimized in the new environment- very badly. ;)

Inspiration	Posted - 2007.05.25 20:03:00 - [211] I now like Jove a little more ;). But seriously, I know how query performance can radicaly differ from one setup to the next. Not to mention with different load characteristics. It is hard to have something thats generic and just works and at the same time having it work well under all possible conditions. Join hints that work miracles on one machine can perform comparitivly unexected slow on another. Identifying and fixing it fast, on a large bussy system is nothing short of excelent. It's fact that these things can't always be spotted and optimized away on anything but the running server. In theory everything is possible ofcourse, but thats for others :). Thanks for your magic and information!
jonfr Gallente	Posted - 2007.05.25 20:05:00 - [212] Edited by: jonfr on 25/05/2007 20:04:39 Why doesn't CCP use QoS to prevent lag ? Under a new name and new look.
Fel Wrath	Posted - 2007.05.25 20:07:00 - [213] CCP should really consider switching to Isolinear chip technology... Hey it worked for James T. Kirk!
Lumy Eve University	Posted - 2007.05.25 20:07:00 - [214] Pardon me, but could you remind me where any of Devs or other CCP employee stated that they were running on MS SQL? IIRC, whole this discussion MS vs. ORACLE started like this: Originally by: Nezz Jaran And out of curiousity, when you say "SQL Server" you are speaking in generic terms, right? You're not actually talking MS SQL Server? Cuz if you mean MS SQL Server, I think I just found your problem (it's all about Oracle on Unix).
Redback96	Posted - 2007.05.25 20:13:00 - [215] All these technical experts and not a single person that knows what they are talking about. :-P
Toramt	Posted - 2007.05.25 20:19:00 - [216] Edited by: Toramt on 25/05/2007 20:21:28 Originally by: Lumy Pardon me, but could you remind me where any of Devs or other CCP employee stated that they were running on MS SQL? IIRC, whole this discussion MS vs. ORACLE started like this: Its has been stated that the forums use the same DB as the game. At http://oldforums.eveonline.com/?a=topic&threadID=19378 we see an error from the forum stating that it can't get to SQL Server. At http://oldforums.eveonline.com/?a=topic&threadID=146596&page=1#3 there is a statement about running SQL Server on Windows 2003 Server. I wouldn't expect them to be that consistent on calling it SQL Server and never slipping in to referencing it as Oracle/MySQL/Postgres/Sybase unless it actually was MS SQL Server running on a W2k3 server. I wonder how MS SQLServer compares to Oracle running with a Oracle TimesTen In-Memory Database cache on the application servers in front of a proper Oracle RAC database cluster.
Inspiration	Posted - 2007.05.25 20:20:00 - [217] Originally by: Redback96 All these technical experts and not a single person that knows what they are talking about. :-P Ok, I will bite by means of a question......do you?
Evil Pookie BURN EDEN	Posted - 2007.05.25 20:23:00 - [218] so we wont be paying for the last 24 hours...... right?
anubis deathbringer Caldari Fluffy Rabbit Killers	Posted - 2007.05.25 20:29:00 - [219] Originally by: Redback96 All these technical experts and not a single person that knows what they are talking about. :-P i fully agree, i personally know nothing and see nothing. life is a big mystery and well personally i think the fact the server was down for 8 hours shouldnt have sparked this large of a topic. give it a rest people. just be glad that they caught it before your character was deleted and/or you lost everything you own
Toramt	Posted - 2007.05.25 20:30:00 - [220] Originally by: jonfr Edited by: jonfr on 25/05/2007 20:04:39 Why doesn't CCP use QoS to prevent lag ? QoS is a method of prioritizing network traffic. Since all the traffic (game packets) are essentially of equal priority to one another, on what basis would they prioritize? I suppose one could try to lower the priority of people who are in station, but they are already using so little traffic anyway that it doesn't seem helpful. The other problem with QoS is that every router along the path needs to support it, or else the benefit is minimal. They can QoS within the datacenter, but setting the 'high priority interactive' QoS bit will not help the packet once it hits someone else's router.

Lumy Eve University	Posted - 2007.05.25 20:39:00 - [221] Originally by: Toramt Its has been stated that the forums use the same DB as the game. At http://oldforums.eveonline.com/?a=topic&threadID=19378 we see an error from the forum stating that it can't get to SQL Server. At http://oldforums.eveonline.com/?a=topic&threadID=146596&page=1#3 there is a statement about running SQL Server on Windows 2003 Server. I wouldn't expect them to be that consistent on calling it SQL Server and never slipping in to referencing it as Oracle/MySQL/Postgres/Sybase unless it actually was MS SQL Server running on a W2k3 server. I'm still not entirely convinced. The first error says only that OLE DB (Microsoft API for connectiong to DB) couldn't connect to SQL server. It could be Oracle (just for as example) as well. And I guess you can run Oracle on Windows server, too. So, unless Devs openly state, what kind of SQL server do they use, the whole discussion about what should they use is pointless (imho).
Pardack State War Academy	Posted - 2007.05.25 20:59:00 - [222] Originally by: Lumy I'm still not entirely convinced. The first error says only that OLE DB (Microsoft API for connectiong to DB) couldn't connect to SQL server. It could be Oracle (just for as example) as well. And I guess you can run Oracle on Windows server, too. So, unless Devs openly state, what kind of SQL server do they use, the whole discussion about what should they use is pointless (imho). It's definately running MSSQL. I'll see if I can find the relevant dev blogs, but it has been stated many times by multiple devs.
Lumy Eve University	Posted - 2007.05.25 21:14:00 - [223] Edited by: Lumy on 25/05/2007 21:19:00 Originally by: Pardack It's definately running MSSQL. I'll see if I can find the relevant dev blogs, but it has been stated many times by multiple devs. Thanks, but you don't have to anymore. I did some googling myself, and yeah, you were right. I should be less lazy. By the way, for all those "split server into two", "use QoS", "use Oracle": look what i have found, before ... well, you probably know
T'ni Iommi	Posted - 2007.05.25 21:32:00 - [224] http://goonfleet.com/open_letter_to_CCP.html
whizzz	Posted - 2007.05.25 23:42:00 - [225] As someone who has worked as a SQL senior developer, as head of operations support and currently as overall head of IT for a large corp, I thought I'd add my 2-penneth for what its worth. 1. 95% of the time a reboot works. When you have lots of servers working together (clustering)and they all start to grind to a halt together, you know it cant be hardware so a reboot (to refresh the software and update error logs is the logical course of action once youve rulled out the communication links). At least the junior DBA had the balls to try and sort out the problem. 2. CCP should pay top dollar for at least 3 elite DBA's to work shifts around the clock. After all its not like they are short of cash and it is supposed to be a 24hr game... and what if the head DBA gets sick... Taking this a step further, why do all the DBA's have to live in the same time zone as everything should be accessible remotely. 3. Neither MS SQL nor Oracle SQL are great products IMHO. Both technologies require lots of computing hardware compared to many of the 'other' sql software providers out there (not that Oracle or Microsoft would ever acknowledge that they exist as one by one they kill them off with their huge marketing spend). 4. CCP run IBM blades using AMD processors. This is the fastest combination of PC server technology - which they are forced into buying due to the number of concurrent users and the processing power required by sql (see 3.). Ask anyone in the industry about blades and AMD processors is that this is not the most stable technology and that Microsoft and IBM don't get along well with each other as they are competitors in many areas. This means new microsoft software patches dont always work well with the ibm and amd hardware adding to the support guys problems. Blades are more bleeding Edge than leading edge. 5. Given its far too late to change your key server infrastructure, I think CCP do an awesome job in keeping things up and running whilst continuing to grow and evolve. I especially would like to thank the DBA for being so upfront and honest in an industry plagued with bull**** and marketing spin.
Derforseti	Posted - 2007.05.26 00:32:00 - [226] It's obviously a broken solenoid. Happens to me all the time.
Manta Avoid	Posted - 2007.05.26 00:53:00 - [227] In reply to whizz - I totally agree. Good on CCP and all their game staff! They didn't make the hardware, so they have to deal with what they can get that suits the job. I'm just extremely pleased that they bought the best H/W they could get! Problems like this always happen tho, so I wouldn't be poking 'em with the rough end of the stick just 'cause there was an issue after a patch release. Sometimes, you just can't be sure what caused it, but you can just as well be sure these guys aren't going to pull an EA on us and just ignore the problems! GG CCP! - Manta
Smash64 Minmatar	Posted - 2007.05.26 00:57:00 - [228] http://digg.com/pc_games/EVE_Creators_CCP_Under_Fire_Again_for_Alleged_Corruption_Open_Letter_Made sad times "All opposition must be stamped into the ground"
Torrstar	Posted - 2007.05.26 11:37:00 - [229] Is the server down now? (7:40 am GMT -5 26/5/2007) Status Indicator on my log in screen shows "unknown" and I get an error message saying that I either haven't established an internet connection (not true), the server isn't running (more likely), or the server address or port number was wrong (I did nothing to change them)
Keten Kennek	Posted - 2007.05.26 11:40:00 - [230] I think TQ is down. I get the same 'unknown' status, and no connect

Aylit Neroon Caldari Caldari Naval Defence Force	Posted - 2007.05.26 12:02:00 - [231] Edited by: Aylit Neroon on 26/05/2007 12:01:46 12 o`clock, and status unknown, I cannot login, either.
Voodoo Mistross Minmatar KickAssCrew	Posted - 2007.05.26 12:05:00 - [232] what is going on?! patch deployed yesterday and extended dt again tonight?!
Xipe Totec	Posted - 2007.05.26 12:11:00 - [233] Maybe they are deploying Revelations 2.0 as we speak ;-) ...
Aylit Neroon Caldari Caldari Naval Defence Force	Posted - 2007.05.26 12:14:00 - [234] Finally, it`s starting up! No offence, but better late then never.
Katiarky Strigo	Posted - 2007.05.26 12:15:00 - [235] Here's my two cents: Eve-Online already lacks scalability in some game sections (fleet battles for example). Don't know how many clusters are there to support the game (e.g.: one cluster for transactions, one for global positioning of ships, another for contracts, etc, etc). Eve-online devs are not keeping up with clients' demands for a more fluid gameplay, so they plan an easy way out, which is top HW aquisitions. Needless to say, that this doesn't solve the problem at all, only delays its effects. These are all short to medium-term "solutions", and there will be a point in time where even HW won't be able to help them. I once saw a movie from one of those fests where devs were talking about the improvements they made, since Eve's first appearance to public, and presented a fleet battle with an enormous amount of ships shooting at each other, with apparentely very small lag. I've never seen such behavior in Eve (I wish I've seen it, but never did). My point is this: DBA admins, MS SQL Servers, Oracles are all already at an very high-level in terms of app development. DBA's are probably making work-arounds and query optimizations that probably have already very small performance gains. Eve's core needs to be reviewed (network, motion predictor, data compression, reduction of queries complexity, data model). And it needs to be reviewed now (if it's not already being done), because we will only see it's benifits in a year or two from today. If it's not being reviewed, I can only hope it can sustain the increasing number of players. But my faith is becoming dimmer...and dimmer.
Inspiration	Posted - 2007.05.26 16:24:00 - [236] Edited by: Inspiration on 26/05/2007 16:24:14 Originally by: Katiarky Strigo Here's my two cents: Eve-Online already lacks scalability in some game sections (fleet battles for example). Don't know how many clusters are there to support the game (e.g.: one cluster for transactions, one for global positioning of ships, another for contracts, etc, etc). Eve-online devs are not keeping up with clients' demands for a more fluid gameplay, so they plan an easy way out, which is top HW aquisitions. Needless to say, that this doesn't solve the problem at all, only delays its effects. These are all short to medium-term "solutions", and there will be a point in time where even HW won't be able to help them. I once saw a movie from one of those fests where devs were talking about the improvements they made, since Eve's first appearance to public, and presented a fleet battle with an enormous amount of ships shooting at each other, with apparentely very small lag. I've never seen such behavior in Eve (I wish I've seen it, but never did). My point is this: DBA admins, MS SQL Servers, Oracles are all already at an very high-level in terms of app development. DBA's are probably making work-arounds and query optimizations that probably have already very small performance gains. Eve's core needs to be reviewed (network, motion predictor, data compression, reduction of queries complexity, data model). And it needs to be reviewed now (if it's not already being done), because we will only see it's benifits in a year or two from today. If it's not being reviewed, I can only hope it can sustain the increasing number of players. But my faith is becoming dimmer...and dimmer. You are entitled to your own oppinion ofcourse. But please realise that the current level of service and gameplay smoothness is waaaaay better then it was two years ago. There is a steady noticable improvement happening, and the extreme surge of new players (or alts) looks like slowing down. To me it looks like you are reading too much from too little information. We can with all due respect only get a superficial idea of all the details you make assumptions about. It can be that certain game aspects like large fleet battles are lagging behind the improvements, i can't know as i never been in one. The problem is being adressed by means of a change of gameplay mechanics. If its gona work, I don't know, players will always try to fight unfair fights and feel great about it, thus blob.....and then blame CCP ocourse! My advice: Cool down :)
Morning Maniac Eve University Ivy League	Posted - 2007.05.26 18:23:00 - [237] Originally by: GM Faolchu Edited by: GM Faolchu on 25/05/2007 04:08:23 The server is currently experiencing database problems, the server team is looking into it. The server should be up and running shortly without any further issues. We apologise greatly for any inconvenience caused and hope that there will be no further issues. Update It looks as though an issue from the previous reboot cropped up again. The server was taken down again to resolve this issue. Anyone who says anything bad about Faolchu will have to deal with me :) Oh yes, server sucks, mucho lag... MM http://eve-ivy.com EVE University commercial
pincalo Caldari The Jin'ei Hizoku Knights Of the Southerncross	Posted - 2007.05.27 00:40:00 - [238] ok since all that i have seen on these posts is noth short of players bashing CCP and DEVs you ppl really have no CLUE how hard these guys work to make TQ run as smoothly as they do, yes they have some problems and bugs but honestly, they are fixing them as fast they can. the techs do know what they are doing and they whole thing you guys need to stop whining about it. ANY game has bugs and glitches we are lucky to have this and if you guys don't have the patience to wait for a server to get back online too bad, crying about it is only going to anger techs, angry techs are not your friends, i know this cause i AM a tech (not for eve). this server set up is the best game i ever played and im sticking with it, bugs and all. and before you reply to this post and say something about me being a kiss, think twice about it because these kind of server set ups are few and far between. i have yet to find a game that sucks me in as much as eve does, and i am here to stay and i won't have a forum argument so read this post and move on cause i will not have a posting war for this thread. if the devs say the bugs will be fixed when they are fixed then i have fulll confidence they will get done, DEAL WITH IT. _______________________________________________ KABOOM!!!!! "what just happened?"
Galdoth Incognito Inc	Posted - 2007.05.27 01:21:00 - [239] Originally by: pincalo ok since all that i have seen on these posts is noth short of players bashing CCP and DEVs you ppl really have no CLUE how hard these guys work to make TQ run as smoothly as they do, yes they have some problems and bugs but honestly, they are fixing them as fast they can. the techs do know what they are doing and they whole thing you guys need to stop whining about it. ANY game has bugs and glitches we are lucky to have this and if you guys don't have the patience to wait for a server to get back online too bad, crying about it is only going to anger techs, angry techs are not your friends, i know this cause i AM a tech (not for eve). this server set up is the best game i ever played and im sticking with it, bugs and all. and before you reply to this post and say something about me being a kiss, think twice about it because these kind of server set ups are few and far between. i have yet to find a game that sucks me in as much as eve does, and i am here to stay and i won't have a forum argument so read this post and move on cause i will not have a posting war for this thread. if the devs say the bugs will be fixed when they are fixed then i have fulll confidence they will get done, DEAL WITH IT. \ signed
Ellisuim Eclipse	Posted - 2007.05.27 06:32:00 - [240] Hello Everyone -Developers and Players, This is the first time I've posted to the forums. First I was confused and mildly amused at the problems the server was having. When I next logged on and saw the post I read through the entire forum discussion. I felt it important to give my opinion as I didn't see it already reflected. I would like to congratulate the CCP. Every systems needs maintenance. Every system will have emergencies. The DBA handled the situation not only properly, but stunningly quickly. This game is played by tens of thousands of people. It is updated with new features all the time. The fact that the problem was identified so quickly and handled so well impresses me greatly. I am amazed that the server is SQL and windows based. But what ever works works. Think about the consequences of the event. No one lost any character data. It was corrected within less than a day. Skill trainings were continued. A warning was issued that allowed time to warp to a safe place. Basically, an update didn't fully function, and was taken offline. The game remained viable stable and playable after a few extra hours offline. If we had this kind of reliability in ALL our services and products we might be warping all over space ourselves! I've always enjoyed EVE, but now I enjoy reading about the CCP company as well. Thank you for your work, time, and caring.

Furion Riddick	Posted - 2007.05.27 13:56:00 - [241] Hmmm, SQL Server, now there lies your problem, tcpm is not good for SQL Server, but Oracle can handle alot of transactions per second (10k/sec) as a single instance, Imagine if it was RAC'd (Oracle on UNIX/LINUX) and on a solid state drive
Ginlol Perkone	Posted - 2007.05.27 17:59:00 - [242] saila system. massive lag doing anything. docking, missions, cans - whatever. sucks bad.
Rasta420	Posted - 2007.07.12 02:04:00 - [243] Originally by: Pelagiad Originally by: Erikel Edited by: Erikel on 25/05/2007 04:27:28 Originally by: Pelagiad And the server is down again or still???? Come on....EVE is 4 F'ing years old and its still in beta? Are you guys drunk or what. CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. I'd like to see a little less pictures of the Dev's drinking and a little more quality put into this product. You guy have an absolute great idea with EVE, bur for 4 years you've *ed away your chances to make it really shine through your lack of concern over stability and quality in favor of features and fluff. BTW who's God Dmn idea was it for the website/forums to share the same database with the game.....kinda makes for more lag and performance issues huh? Now i know why they took the server down a second time. They are messing with that arsehat. So you like the way the game is working right now then? Just being clear, when would you say was the last time Eve was working the way it's supposed to be working, as communicated to us by CCP? Dude common, you have to admit that this game is awsome, or you wouldnt be taking ur time to ***** and complain about it. Personaly for me, this game is one of the only true SciFi MMO spaceship based games that truely goes into the depth and customization that I have looked for in other MMO's. And as far as this beta is concerned, I think the games "lag" is quite low compaired to other games such as WoW and especially Guildwars, now that is Lag City especially if you on wireless DSL.
Acacia Everto State War Academy	Posted - 2007.07.12 02:09:00 - [244] Originally by: Rasta420 Originally by: Pelagiad Originally by: Erikel Edited by: Erikel on 25/05/2007 04:27:28 Originally by: Pelagiad And the server is down again or still???? Come on....EVE is 4 F'ing years old and its still in beta? Are you guys drunk or what. CCP you do know that we all PAY for this crap right.....get professional....make the ** work.....do your damn job right. Do you guys even have a quality assurance dept? You know a team of people who make sure the crap your drunken dev's put out is QUALITY. I'd like to see a little less pictures of the Dev's drinking and a little more quality put into this product. You guy have an absolute great idea with EVE, bur for 4 years you've *ed away your chances to make it really shine through your lack of concern over stability and quality in favor of features and fluff. BTW who's God Dmn idea was it for the website/forums to share the same database with the game.....kinda makes for more lag and performance issues huh? Now i know why they took the server down a second time. They are messing with that arsehat. So you like the way the game is working right now then? Just being clear, when would you say was the last time Eve was working the way it's supposed to be working, as communicated to us by CCP? Dude common, you have to admit that this game is awsome, or you wouldnt be taking ur time to ***** and complain about it. Personaly for me, this game is one of the only true SciFi MMO spaceship based games that truely goes into the depth and customization that I have looked for in other MMO's. And as far as this beta is concerned, I think the games "lag" is quite low compaired to other games such as WoW and especially Guildwars, now that is Lag City especially if you on wireless DSL. Necrobump ftl

Yipsilanti Forum Moderator Interstellar Services Department	Posted - 2007.07.12 02:10:00 - [245] This conversation has been over for quite a while, please do not resurrect old threads ___ forum rules \| [email protected] \| Our Website



Pages: 1 2 3 4 5 6 7 8 9 :: [one page]

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.