EVE Search - The current server issues

All Channels

EVE General Discussion

The current server issues

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: 1 2 3 4 [5] 6 7 8 9 :: one page
Author	Thread Statistics \| Show CCP posts - 9 post(s)
Xrious	Posted - 2007.05.25 10:15:00 - [121] Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? I apologise and know that you need sleep, but employ someone else. Thats really what we're paying for. We all abuse M$, but I had alot less problems with SP2 than CCP seems have with their own systems. Sorry, just had the message pop up on my screen telling me that the server is shutting down at 11am for its normal reboot. Very very anoying when you've spent hours getting in. Kinda rubbing salt in the wounds there CCP. X Sig? what sig?
BCBArclight Odessa Operations Sylph Alliance	Posted - 2007.05.25 10:21:00 - [122] Originally by: Xrious Edited by: Xrious on 25/05/2007 10:15:52 Does anyone find it worrying that a junior DBA appears to have the authority to shutdown a server with 30,000 users on? Sure only the person at top should have that authority? Not too uncommon, some companies have procedures that need to be followed when certain conditions arise. These need to be done asap to either save data or minimise effect on other systems, by whoever is there at the time. Its possible this was the case or the junior DBA wanted a laugh and pressed the big red button Odessa Operations are Recruiting
Mortim Minmatar Madison Industrial Co. Sylph Alliance	Posted - 2007.05.25 10:24:00 - [123] Aside from any complaints that have already been fronted, I'd like to do two things. Firstly, offer my thanks, EvE Online is nothing short of genius, the fact that the demands, and usage is wild, unstable, and always looking to change can't make keeping anything like this stable easy in the slightest. The hour maintenance time is far better than the constant problems that other, less innovative games suffer. My salute to your efforts over, my second point was that you always offer some form of explanation. Numerous people would be content with a simple "It's working", but the involvement with players (of which Im guessing a good number are of a computing nature) is great. So my question - Is there any way in which we can be more involved with the problems encountered, disasters averted and thought that goes into maintaining and upgrading EvE at all? A server team blog or some such would be ideal (pretty sure one doesn't exist). I'm personally starting my career in Systems Development, and insight into the backbone of something Im very familiar with would probably prove to me it's not all "magic". Mort
Devious	Posted - 2007.05.25 10:37:00 - [124] How many games have you played that are 3-4 years past beta where the dev's and support staff tell you what went wrong and what was fixed or will be fixed??? i can count only one. EVE Online.
Ifni Applied Eugenics	Posted - 2007.05.25 10:37:00 - [125] I love the way loads od people criticise CCP for their choices So far mud has been slung at CCP, Microsoft, the junior DB admin, the hardware, and at each other. The people at CCP know their setup better than anyone else, and when a problem like todays arises, there are certain procedures in place to deal with it. It is only in exceptional circumstances that a problem gets elevated to the point where you wake up the guy above you in the food chain, as can be seen by tonight's activities. Yes, it sucks, yes, you feel hard done by, but the EULA that you all agreed to states that the performance of the game may vary. As has been said before certain problems simply can not be foreseen prior to a complete roll out. Hell, why do you think Microsoft, the largest OS manufacturer in the world, rolls out constant updates and patches? And they've been doing it for decades. So quit your moaning, and rather than belittle the guys for doing their jobs, congratulate them for keeping the server running for the other 99.9% of the time. <3 Valar, <3 junior DB admin. You take what is offered. And that must sometimes be enough.
Marakish Kalar Minmatar Blancmange Interstellar Kilts Archaean Cooperative	Posted - 2007.05.25 10:42:00 - [126] Edited by: Marakish Kalar on 25/05/2007 10:43:08 Well done guys good to see the duty admin did what they could, but then decided to call out the guy who gets paid millions to do his job.... When I was younger this whole episode would have had me pulling my hair out complaining and B B Biting my tonge. I'm glad to say that now it doesn't, does? d..? What was I saying? zzzzzz Keep up the good work guys, and thanks. . Marakish Kalar [BLANC]MINMATAR'S EDGE
Xrious	Posted - 2007.05.25 10:54:00 - [127] Edited by: Xrious on 25/05/2007 10:54:06 I'm sorry, I'm not aiming to CCP bash, but the stability of the system isn't good enough IMHO. SQL server's don't need to be rebooted everyday, they stay up for years, not 23 hours. As someone else pointed out, running the forums and the website on the same cluster as the game? great, game server goes down, the players can't even find out about it. Someone said that how often does a company give explanation's on why a system screwed up 3-4 years out of beta, no offense but a system shouldn't be having these kinds of problems 3-4 years out of beta. Giving explanation's on what has happened is great, we really do appreciate it, but I think the player base demands it. Eve is an intelligent, strategic game, not a reflex FPS and the type of player tends to be more mature (well, mostly ) that you'd find on other games out there. A vast percentage of the player base understands the technology behind Eve so they could hardly try to cover it up. IMHO if there was an alternative to Eve out there, then I'm sure things would be different. People would have a choice, people would vote with their wallet. As it is, Eve is a one off, its unique so there's isn't anywhere else we can go for our 'fix', but it doesn't mean we need to be quiet about it. Anyway, just my 2 cents. Hopefully people at CCP are reading this. If nothing else gets done, could we at least have more information when things like this happen? 4 hours ( I think I read) bewteen updates is perhaps a little slack. Keeping the user's informed costs you little, but makes a massive different to us. Thanks X Edit: typo
Ethaet	Posted - 2007.05.25 10:55:00 - [128] Originally by: BCBArclight Originally by: CCP Valar This causes extreme CPU load on our SQL Server and is not easily identified in a database trace due to the long duration of the proc(as in hours) when this happens. When the junior database admin first looked at the problem he did not identify the real cause and thus the first reboot did not fix the problem. Good old SQL server, if you need someone else to throw stuff at when it breaks I need a job after finishing uni now I worked in SQL support for a year, I have lots of ways to mess it up that havent been tried yet SQL servers mess themselves up without need for a human fairly regularly :P :D
Rekindle	Posted - 2007.05.25 10:58:00 - [129] so many back seat admins. Kinda late in the design to be whining about SQL isn't it? (besides SQL is a great product). -------------------------------------------
Digital Anarchist	Posted - 2007.05.25 11:01:00 - [130] Originally by: CCP Valar The issues tonight were caused by the contract system. The original design for contract lookups never intended that you could look up contracts in other regions and because of that and other issues with contracts, the query plans the SQL Server creates for the contracts lookup can go out of whack. Dear CCP, please don't nerf the global view of contracts because of this. ---------------- Nerf government!
Suzii	Posted - 2007.05.25 11:02:00 - [131] are guristas supposed to drop only ammo 90% of the time
Aleksandr Cirtus Caldari Stronghold corp Curse Alliance	Posted - 2007.05.25 11:08:00 - [132] Edited by: Aleksandr Cirtus on 25/05/2007 11:07:21 Personally, I would like to thank CCP and the dev team, and the GM team, all at the same time. True, the issues were starting to get annoying, but they identified and rectified the problem in short order. In addition, a GM was able to assist with a side effect in one of my missions within a minute. A MINUTE. EVE has come a long way in four years, and I would like to see all the nay-sayers pause and reflect on past experiences and dare to tell me things are getting worse. Cheers to all! Aleksandr Cirtus Senior Knight and Ghost Fleet Pilot of Stronghold Curse Alliance
Oratu Brutor tribe	Posted - 2007.05.25 11:08:00 - [133] lol, nerds. Pre deployment testing ftw !!!! Doh! iam a nerd too :(
Tiger Ma	Posted - 2007.05.25 11:15:00 - [134] I run a database consulting company and basically make my money fixing databases and making them go faster. You're probably now expecting me to start ranting about MS SQL Server, junior DBAs and OMG where's my compensation for lost training time. Er, no. I do have to wonder if a clustered MS SQL Server farm is the best way to go, not out of any dislike of MS SQL Server (does a great job when used appropriately; I know large banks that use it very successfully for systems that with respect are considerably larger and more complex than EVE)... but because of the hardware constraints any MS architecture imposes. What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. But I can't see CCP being willing to walk away from some serious hardware investments to change their fundamental architecture. (Although think about solid state devices for all your tempdbs; it gets you past the 16Gb memory limit without architectural changes.) SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. An obvious win CCP are already discussing in public is separating the web database from the game database. That's really a no-brainer and it will have disproportionally good effects as the main DB won't have to cache web-related data or query plans anymore. I'd suggest a third database (on dedicated hardware) for character info is a simple next step. I'd even wonder how much of the game database could be distributed... the contracts system is pretty self-contained, for example, with nothing outside contracts obviously depending on it. Anyway, CCP, you have some good talent in your player base. I and many others would be absolutely delighted to volunteer under any reasonable NDA and T&C. If I'm allowed to use you as a reference I'll do it for free. 8-) Lastly, I've totally been the one who had to bring down a production environment on no notice. I understand why it has to happen. Good call on taking an unpopular step to get the job done. (And let me know if you want a hand query tuning!)
Armoured C	Posted - 2007.05.25 11:15:00 - [135] Edited by: Armoured C on 25/05/2007 11:15:38 great work on getting the problem sorted to all those moaning at CCP and dev team and server admin and what ever here a question for you try running your own server capabable of running 30k pilots in one go then we shall see how much we could moan at you every other game you have to log into a different server to do either missions , pvp and what not this is the only mmo i know of where it is all intergrated so shut up with you moaning do you kniw everything about there servers or anything about there set up at all? have you managed a 30k plus pilot server with intergrated website and forum? if any of the awnser are no and you still want to moan just shut up, they are dong the best they can in any situation you get my personal thanks CCP it could of been worst and been out for a week but servers were only down for a few hours if you want to go and moan at CCP go to WOW forums and join there club
Gaven Blands	Posted - 2007.05.25 11:17:00 - [136] The lag was perfectly timed. It got me a podkill that I shouldn't have been entitled to. And a free Vexor he ejcted from. If I could show you all the relevant GM tripe, you'd see why this podkill was so poingnant, what with it belonging to a macromass of paid accounts that can't be found guilty of exploiting under circumstances that singular paid accounts can be. I expect to found guilty of causing that lag last night and banned quickly for commiting the heinous crime of harrassing the macromass. Eve: Cheats prosper.
Fryke Caldari CRESCENT Inc.	Posted - 2007.05.25 11:20:00 - [137] As a programmer I know how you feel and I'm actually pretty happy you fixed it up as soon as you did. P.S. Also I'm curious why you have to reboot everyday. Why not once a week? Or even better, once a month?
Vergil 577	Posted - 2007.05.25 11:21:00 - [138] Originally by: Oratu lol, nerds. Pre deployment testing ftw !!!! Doh! iam a nerd too :( First Oratu welcome to the club Second it's amazing how people programed all of this I'd get a heart-attack and die... and I'm not even old enough for that and the amount of time they fixed the problem in is also good. only bad thing is I'm in the wrong time zone +10GMT makes DT fall on 9pm... but otherwise this is impressive running a whole universe with lots of solar systems. People will always complain there is nothing you can do about it, but you can't let it get to you. Live on EVE-Online.
Armoured C	Posted - 2007.05.25 11:30:00 - [139] wow you lost you pod big deal it only a game, just get on with you eve life and work your way back up, do you think they would of done this intentionally or something to personally spite your pod to death because they hate you that much? i am no programme i am a aeronauticle engineer so i dont know what half of you are on about but i am sure they did what they did because it was necessary to keep the server going, i lost a ship in the lag to but i rather they closed the server to stop the lag rather than keep it open and sit in a station doing nothing while i cant even type a message in the type box, and i also rather they do the matainince a hour a day than let say 1 day a week or 2 days a month or what have you least your only a hour away from game time instead of waiting a intire day but even if CCP did that some of you would still probably moan anyway
Stephar The High Priest	Posted - 2007.05.25 11:31:00 - [140] Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's.
Armoured C	Posted - 2007.05.25 11:37:00 - [141] Originally by: Stephar Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's. lol i loved to see them work there for a day and then see how much they get slapped and proven wrong ...
The Power HighTech Marines Ltd. FREGE Alliance	Posted - 2007.05.25 11:53:00 - [142] Edited by: The Power on 25/05/2007 11:59:19 Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. Oh how I like this ... We have alot of ppl running around blaming CCP for not having the servers up. And now they start the religion war between Unix and MS. Sure MS can run large servers .... http://h20341.www2.hp.com/integrity/cache/342431-0-0-0-121.html I'm DBA on an MS sql server set-up of 3 x 16 cores each with 32 gb ram. I am not saying MS is the only way. hell no .. but sometimes a company has to choice the platform they'll run on based on the kind of ppl thay can hirer I dont want to go into details about CCP job and how they are doing it.
Neyland	Posted - 2007.05.25 11:55:00 - [143] Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. Windows 2003 R2 Datacenter 512 MB of RAM required; 1 GB of RAM recommended; 128 GB maximum for x86-based computers; 2 TB maximum for x64 and ia64 computers Maximums: 64-way capable multiprocessor machine supported for 64-bit processors and 32-way capable multiprocessor machine supported for 32-bit processors http://technet.microsoft.com/en-us/windowsserver/bb430827.aspx
Gaven Blands	Posted - 2007.05.25 12:02:00 - [144] Originally by: Armoured C omgwtfbbgreadingcomprehensionftwlolbbqoneoneelevenbbqnerfthedrake In the words of the late great Bill Hicks.... "Well........... looks like we have ourselves a reader" Later, while I was hanging from the tree..... Eve: Cheats prosper.
Dark Flare Caldari Corpus PCG The State	Posted - 2007.05.25 12:06:00 - [145] Originally by: Nezz Jaran Edited by: Nezz Jaran on 25/05/2007 09:00:20 Cool, so the server farm is "now fixed" and it's time for me to go to bed (1 am). Glad I had a chance to ******* play tonight. Don't like playing when the servers are bouncing around, too much of a chance for something to go wrong. sigh Think of how LITTLE unscheduled downtime there is, and how much work goes into this game from CCP, then re-evaluate your little whine about missing one nights play. Good job CCP, you're doing some excellento work! I manage servers at work, and I sure as hell wouldn't want to have to manage as much, and as complicated as you do. Keep it up Dark Flare - Corpus PCG
Defiant952	Posted - 2007.05.25 12:10:00 - [146] You know, All of you complaining about the server stability and crap, How often are there problems like this? I mean come on very rarely does this happen and as to the "four years old it shouldn't happen" How many games do you play that support 30k+ simultaneous players AND the new content and features EVE puts out? You should be thanking them for the rare problems like this they have all things considered. Anyways, Keep up the awesome job CCP don't let the ones that do nothing but whine bother you.
Cryten Jones Gallente Infinite Development Consortium	Posted - 2007.05.25 12:11:00 - [147] Originally by: Tiger Ma SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. The EvE cluster used solid state disks already mate. Please see the following links and quotes. http://myeve.eve-online.com/devblog.asp?a=blog&bid=303 http://www.superssd.com/products/ramsan-400/ Quote: The new Tranquility consists of 70 x IBM LS20 AMD Opteron Blades, each Blade sporting two 2.4 GHz 64-bit Opteron processors with 2-4 GB of RAM, depending on application. That's a 140 CPUs, a full replacement of the current 140 Intel XEON 2.8 GHz CPUs. Hope that helps. -CJ
Saturous Viper Squad Triumvirate.	Posted - 2007.05.25 12:13:00 - [148] Horrah, the servers up'ish. Stop moaning and get your moneys worth whilst you still can. It makes me laugh how people ***** all night about it and once its up they still go on about it and waste more of their time. Yeah its hard to run a stable server for 30k-ish players at a time. My only guess is anyone who complains about 'lag' and 'disconnecting' has never played other MMORPGS and such. They all do it. "The key to success is the ability to put up with failure." - Follow that and everything will be fine.
Lowa Gallente North Star Networks Cruel Intentions	Posted - 2007.05.25 12:16:00 - [149] Wooohoo! Nerd discussion! I'm SO in! First, to all the people yelling about SQL vs Oracle etc. Get clue or shut it. <3 [Nerd mode] Originally by: Tiger Ma What's the most memory you can put on an Intel box these days, 16Gb? In database land that really isn't very much. And the maximum number of cores, 8 dual cores? Heck, even 8 quad cores doesn't look that impressive next to cheap offerings like a Sun T2000 with 4 processors... each with 32 cores. And you can run Linux on that. True. But if the application does not scale over all cores then you can have a 100 and it wont matter. (Not saying it doesnt but hardware as you know only account for so much) And as you said, CCP aint going to change their investment. But hell it would be cool to see what the difference would from be a pure tech-geek possition. Originally by: Tiger Ma (Although think about solid state devices for all your tempdbs; it gets you past the 16Gb memory limit without architectural changes.) Just as an FYI, they have been running Texas Memory systems SSD's for quite some time. And recently upgraded them from 64GB to 128GB each if I remember correctly. Each of those can do like a gazillion IO's with "pretty good" response times. I believe bookmark-tables (before removal), market, inventories and some other things are placed on those beasts. Is that the salvation of all? No, but it sure does help. Originally by: Tiger Ma SAN is good but most benchmarking shows it has worse latency than DAS (even though the overall throughput goes up) - consider local disk on nice 15k rpm SCSI for the things where you just gotta have the answer now. No. Plain and simple no. In comparision with the high-end storage systems a single disk or DAS will look like a squashed snail. And in comparision with the SSD systems...Just mentioning the IOs handled by the SSD would make any physical disk explode. What you need to think of is a couple of things. 1. This is an extreeeeeeme multi-threaded environment (Obvious, I know) and the word "random access" takes on a whole other meaning A normal disk, even a high-end storage system, can not come close to handle the requests EVE generates (Damn players!). If you take a look at www.storageperformance.org you'll see that even the fastets systems would a) not stand a chance or b) cost an enourmous amount of money (And still be beaten by the SSD system who also are tested) 2. A single disk can look good in a single threaded test with low IO. Take into consideration that the fastets disks today can generate about 300 IO's (Vendor saturation tests can show close to 400) and in a storage system the fastest will give you about 250 IO's / disk. The reason for seing high numbers on DAS or single physical disks sometimes normally comes from two things: 1. IO's are cached in the server 2. IO's are cached on the physical disk (a HUGE risk of corruption comes into play there and its a big NO NO to use!) Now, that means as long as you short-stroke the disk, only run like 2k (or even 512b) IO's you will see 300-400 IO's and very few real applications generate those kind of IO's. Hell, both Oracle and MS SQL shoots out 64k or even 128k IO's and that is either by DB design or forced by the file system. You could go RAW on Oracle for example but Oracle them selves hardly promotes that anymore. I dont pretend to fully understand the EVE environment (and trust me I've tried) but I have a pretty good idea. I do know that its so many different things going on that there is no single thing that will fix all problems. [/Nerd mode] Sorry about the rant, its not directed to you Tiger specifically, I just saw my chance to talk nerd stuff about something I actually have a black belt in. :) Cheers, Lowa What if the truth was something else?
nomis ekoobmah	Posted - 2007.05.25 12:24:00 - [150] Originally by: Stephar Thanks for the detailed explanation on what happened. I did notice that contracts were especially nonresponsive. And ignore the armchair programmers... well at least the ones who are being critical. I doubt any of them have worked on a cluster the size of EVE's. Does a 4096 node Linux cluster count? I used to write massively parallel applications for an oil exploration company.

Pages: 1 2 3 4 [5] 6 7 8 9 :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.