EVE Search - Constant Start-Up Issues: Poor Practice

Check My IP Information

All Channels

EVE General Discussion

Constant Start-Up Issues: Poor Practice

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: [1] 2 :: one page

Author	Thread Statistics \| Show CCP posts - 2 post(s)
JForce Black Lance Against ALL Authorities	Posted - 2007.03.19 13:01:00 - [1] Of course as soon as I say the words "I work in I.T." I'm bound to be flamed. But I do And funnily enough I work in an area responsible for any changes, and the required release of that change, into the production environment. What I find unusual is the number of startup issues that occur due to changes made. Firstly, most are changes the community isn't told about. We know to set a long skill on patch day, and to be patient till it comes up. We know that it won't come up when you say it will. That is normal. However when you make changes, ANY changes to your production environment, you are taking a risk. And it seems to me that your risk-assessment procedures are poor at best. It's about managing expectations. We expect Tranquility to start up again at 12. Why? Because you haven't messed with it at all. If you're going to mess with it, in any way, then TELL US. That way we can expect to have an extended outage. I'm not going to comment on refunds etc, because updates are needed and outages happen. However CCP IMHO have poor change and release processes, and poor management of customer expectations around them.
Dravin Dread	Posted - 2007.03.19 13:07:00 - [2] Yep, just a litle CYA is all that is needed. CYA always a good idea when making changes.

Timmeh 2000 Forum Moderator Interstellar Services Department	Posted - 2007.03.19 13:07:00 - [3] This topic is already being discussed here. forum rules \| [email protected] \| Our Website! "Formerly known as MiniMeph" \| Do you Role Play in Eve?


Timmeh 2000 Forum Moderator Interstellar Services Department	Posted - 2007.03.19 13:47:00 - [4] bump Topic re-opened. forum rules \| [email protected] \| Our Website! "Formerly known as MiniMeph" \| Do you Role Play in Eve?

Melissa Ravenflame Caldari Aquae-Sulis	Posted - 2007.03.19 14:00:00 - [5] I suppose a lot of it involves having a representative architecture to test changes on. As far as I know (prepare flame retardant material ) CCP do not have a server farm as their pre-production environment which is an exact copy of production. As such, putting in a change, especially a hardware change, does leave them open to problems. The alternative is a very expensive, mostly redundant duplicate test system. However, I've done the Tech Arch for a number of clients which make very good use of that test system so it is possible
Fork Boy	Posted - 2007.03.19 14:02:00 - [6] I'm an old timer system admin too, and there are a few things I'd like to say. 1) Sharkbait showed a short while back that good communication is a great key to keeping your user base calm, even when things aren't working. 2) A little advanced warning would be appreciated. Just mention it to us that something big is going to go down. Probably best not to try to slip it in on the quiet he day before an announced big patch day. 3) Sadly, no matter what pre planning is done, it's not always possible to predict the behaviour of servers under load on a production system. Unless CCP can have an exact mirror of the TQ cluster, same size, same makeup, same simulated load. (At double the cost for subscribers?). We've gone live with a good few rollouts that have not behaved anything remotely like they did under our tests, when we extrapolated load / users etc. I really can't see what more CCP can do in reality, other than let us know what they are planning to do, and keep us up to date when it starts to go wrong. ================================= A fork is a cold, shiny tool To pierce, tear and ingest. Whoever has the fork in hand Controls the meal of their choice.
Jastra Gallente Gallente Venture Inc	Posted - 2007.03.19 14:27:00 - [7] Originally by: JForce Of course as soon as I say the words "I work in I.T." I'm bound to be flamed. But I do And funnily enough I work in an area responsible for any changes, and the required release of that change, into the production environment. What I find unusual is the number of startup issues that occur due to changes made. Firstly, most are changes the community isn't told about. We know to set a long skill on patch day, and to be patient till it comes up. We know that it won't come up when you say it will. That is normal. However when you make changes, ANY changes to your production environment, you are taking a risk. And it seems to me that your risk-assessment procedures are poor at best. It's about managing expectations. We expect Tranquility to start up again at 12. Why? Because you haven't messed with it at all. If you're going to mess with it, in any way, then TELL US. That way we can expect to have an extended outage. I'm not going to comment on refunds etc, because updates are needed and outages happen. However CCP IMHO have poor change and release processes, and poor management of customer expectations around them. The underlying and risk management/mitigation processes may be spot on, they key element missing is communication, though perhaps asking CCP to make public their change schedule is probably asking a bit much since most people will still post here and say "why isnt the server up" :D That being said, you work in IT but how many customers would you notify if a server was getting an upgrade during down time ? - admittedly the window is small here but even so, you might let the IT management chain know, maybe some team leaders that there is a risk, but would you tell everybody ?
Cadela Fria Amarr Eye of God X-PACT	Posted - 2007.03.19 14:37:00 - [8] Yes well, just about everyone in EVE is a Systems admin, game developer, server manager and/or work with the same things CCP do, and somehow everyone always knows better then CCP. Atleast thats the same argument everytime - "Hi I'm joe schmoe and I'm computer smart in one field or another, so therefor I unquestionably know what I'm talking about - This means you can't tell me I'm wrong...that being said, CCP is all wrong about how they do things and should bow their heads in shame and follow my every word! BOW DOWN!" Knowledge is a priviledge, not a right
Ms Freak Amarr NCN Corp Chaos Incarnate.	Posted - 2007.03.19 14:41:00 - [9] Originally by: JForce Of course as soon as I say the words "I work in I.T." I'm bound to be flamed. But I do And funnily enough I work in an area responsible for any changes, and the required release of that change, into the production environment. What I find unusual is the number of startup issues that occur due to changes made. Firstly, most are changes the community isn't told about. We know to set a long skill on patch day, and to be patient till it comes up. We know that it won't come up when you say it will. That is normal. However when you make changes, ANY changes to your production environment, you are taking a risk. And it seems to me that your risk-assessment procedures are poor at best. It's about managing expectations. We expect Tranquility to start up again at 12. Why? Because you haven't messed with it at all. If you're going to mess with it, in any way, then TELL US. That way we can expect to have an extended outage. I'm not going to comment on refunds etc, because updates are needed and outages happen. However CCP IMHO have poor change and release processes, and poor management of customer expectations around them. Good for you!! I'm a developer and agree that CCP change control is somewhat "pap" but theres no need for the whole "Wow i work in IT so i must be right" thing. I'm sure CCP know thier processes/methods/software/coffee machine is not great or working properly. Leave em to sort thier own s**t out.
Akita T Caldari Caldari Navy Volunteer Task Force	Posted - 2007.03.19 14:43:00 - [10] What the OP is saying is that the proxy server change on today's DT should have been announced in a news item a couple of days ago, alongside the announcement for the patch on the 20th. That, and only that. But in a lot more words. _ MySkills \| Module/Rig stacknerfing explained

Miss KillSome	Posted - 2007.03.19 15:17:00 - [11] just gives us some info when u are planning on doing something to TQ that could cause problems. That is hardware upgrades, small software upgrades, anything.. One day ahead i think u know that u are going to add some hardware, its not like its a 10 minute ahead decission..We dont want exact numbers, if u are afraid of some buisness leak (though it sounds fine that u are adding 10 more Blades to the whole system..). Just keep us informed on stuff, it doesnt take u even 10 minutes u write appropriate notice in news..
Jarphen Bard Gallente	Posted - 2007.03.19 15:51:00 - [12] I agree with the OP completely. Have the downtime, got to, no question, but just let us know if there are changes planned in the downtime so we can do a little risk management of our own. De motu corporum in gyrum
JForce Black Lance Against ALL Authorities	Posted - 2007.03.20 01:26:00 - [13] Thanks to the mod team for re-opening my thread And yes of course CCP are doing what they can as far as lessening the risks etc. But they should know better than everyone the environment they're dealing with, and by that I mean the gaming one. One poster asked if I would notify every person if were doing a minor hardware upgrade etc. The answer to that is that yes we would. Every time. We have to. But it's not a valid comparison....because only about 25% of our systems are 24/7. So if we make a change to a system outside of its agreed operating hours then it's a 0 impact change. We all know that Eve players are insane people. We're them. Every single day there are thousands of people sitting around waiting to log in after DT...to play. To change a skill. To complete a market order. Then go to bed/work/out. The best analogy I can think of is that you own a restaurant, a very popular restaurant. Your restaurant opens each day at 12. Every day hundreds of people line up outside to come in. If you know that one day you're having the dining room re-painted, and that the plan is for the painters to finish by 12, but there's always the chance for it to go longer...then you'd tell people. You'd put a sign up a few days before. So your regular customers wouldn't line up outside at 11:45. Maybe they'd plan to eat elsewhere that day, or they'd drop by an hour later. Manage expectations CCP. We still love you, but take note of what is some nice constructive feedback in this thread
Cadela Fria Amarr Eye of God X-PACT	Posted - 2007.03.20 01:54:00 - [14] Why whould they? If you're telling me they don't already know these things, then I laugh. Of course they know all this. It's just like I said in my last post. For some reason everybody is a darn expert and knows better then CCP these days. It's just friggin silly. No offence intended Knowledge is a priviledge, not a right
Viktoria Maher	Posted - 2007.03.20 02:01:00 - [15] Clearly not, as the change was not announced beforehand. ------- Hull tanking? Pffft real men pod tank - Godar Marak
JForce Black Lance Against ALL Authorities	Posted - 2007.03.20 02:13:00 - [16] Originally by: Cadela Fria Why whould they? If you're telling me they don't already know these things, then I laugh. Of course they know all this. It's just like I said in my last post. For some reason everybody is a darn expert and knows better then CCP these days. It's just friggin silly. No offence intended Well the problem is that there ARE people here who are experts at some things. That's not to say that CCP aren't. But I can confidently tell you that CCP aren't following industry best-practise with this. No one says they have to. But that's why you have to endure moaning thread after moaning thread when DT goes long cause an non-advertised change goes wrong.
Lokyi	Posted - 2007.03.20 05:55:00 - [17] Just like I posted in one of the many other threads about this topic (and I think this is actually the most coherent one I've read yet), I think CCP should make sure that at least ONE of these 100 new employees they're advertising for should be given the job of alerting the community to any possible modification being made to the server, either hardware or software. That way we can at least expect something to go silly when the server restarts. Well, expect when it's more likely to happen than normal anyway :P
Kylar Renpurs	Posted - 2007.03.20 06:02:00 - [18] With myself working in IT too, and although I don't diss best practice (though it is a pain), there's one very simple answer. Deployment issues. It doesn't matter how many times you test and re-test in development, when you go live it's murphy's law that there's always some problem which can't be anticipated.
Telemicus Thrace Thrace Inc Ushra'Khan	Posted - 2007.03.20 07:03:00 - [19] For those of us whose prime time is just after DT it is a major issue that seems to be geting more and more frequent. The lack of warning is the real killer, especially in this scenario where we lose Monday evening without warning when many of us were running around to fix stuff up before we lost Tuesday evening to the patch. If I'd known Sunday night I wouldn't be able to log in until Wednesday I would have done a fair few things differently. Contracts, POS fuel, skills. All these things run persistantly to the good or ill of the players. Under those circumstances there should be some warning if I'm going to be cut off for days at a time while others can still come along and blow my stuff up while I'm at work. I understand I'm stuck with mandatory 2 weeks DT a year because of my timezone. I understand that every time a patch comes out I lose a day. What I would like to see is all these little upgrades that routinely bugger up any logistics / operations / contracts / skill changes I have planned rolled into one upgrade. Take the servers offline all evening once a month if you have to and warn us in advance. That would be better than these random blackouts. I know I can't have a level playing field due to my timezone but cut us some slack. Just restrict these upgrades to designated patch nights. >> RECRUITING <<
JForce Black Lance Against ALL Authorities	Posted - 2007.03.22 13:14:00 - [20] Once again we see startup issues, affecting thousands of pilots. Problem Management time? How are these issues investigated and tracked I wonder?

heheheh Singularity.	Posted - 2007.03.22 13:17:00 - [21] Edited by: heheheh on 22/03/2007 13:14:08 Mummy mummy i want it now !!
Presidente Gallente	Posted - 2007.03.22 13:20:00 - [22] Edited by: Presidente Gallente on 22/03/2007 13:16:48 Should I tell you something: I can understand BUT I can live with it. And you should LEARN to. EVE is not the center of my reallife. I pay EUR 10,00 per months which means: 33 euro cent a day. I am working with hardware since 20 years. I know all the simple to complex problems I have with a single PC. So I understand that CCP is dealing with state of the art concepts to make EVE happen. From this POV I see no reason to have the right to demand something because I pay simple EUR 10,00 per month. Let them do their job, be patient ... CCP will not leave us alone. And: an unexpected downtime is a great advantage to let me cool down and concentrate on RL stuff. OVER! OUT! Pres
JForce Black Lance Against ALL Authorities	Posted - 2007.03.22 13:22:00 - [23] Originally by: heheheh Edited by: heheheh on 22/03/2007 13:14:08 Mummy mummy i want it now !! Why dont you try reading the thread? Perhaps then you'll see there's no whining, no "fix it now", just a civil discussion on the way CCP manage their DT startup problems. Try not posting straight out of your ******* next time.
JForce Black Lance Against ALL Authorities	Posted - 2007.03.22 13:23:00 - [24] Originally by: Presidente Gallente Edited by: Presidente Gallente on 22/03/2007 13:16:48 Should I tell you something: I can understand BUT I can live with it. And you should LEARN to. EVE is not the center of my reallife. I pay EUR 10,00 per months which means: 33 euro cent a day. I am working with hardware since 20 years. I know all the simple to complex problems I have with a single PC. So I understand that CCP is dealing with state of the art concepts to make EVE happen. From this POV I see no reason to have the right to demand something because I pay simple EUR 10,00 per month. Let them do their job, be patient ... CCP will not leave us alone. And: an unexpected downtime is a great advantage to let me cool down and concentrate on RL stuff. OVER! OUT! Pres However if you had a dozen POSs to refuel and defend and manufacturing to deal with and things like that, you might expect to come on and play. And if you weren't able to, to be told in advance when they make changes that could go wrong.
50freefly Caldari Purify Namtz'aar k'in	Posted - 2007.03.22 13:25:00 - [25] Originally by: Presidente Gallente Edited by: Presidente Gallente on 22/03/2007 13:16:48 And: an unexpected downtime is a great advantage to let me cool down and concentrate on RL stuff. Obviously not as you are posting on this thread! Oh btw hi =) Originally by: Eight Ace For reasons that have been lost in the mists of time all caldari ships are designed by two people. One does the left hand side and the other does the right. And they never meet.
Barbarellas Daughter Lonely Barbarella	Posted - 2007.03.22 13:33:00 - [26] i work in IT too. but i dont have anything to do with MMO servers. so i stfu.
Presidente Gallente	Posted - 2007.03.22 13:35:00 - [27] Originally by: 50freefly Originally by: Presidente Gallente Edited by: Presidente Gallente on 22/03/2007 13:16:48 And: an unexpected downtime is a great advantage to let me cool down and concentrate on RL stuff. Obviously not as you are posting on this thread! Oh btw hi =) Hehe ... damnit ... I am suffering from withdrawal syndromes and you smashed the truth in my face! An alcoholic would now lie to himself: forum IS rl stuff. I will call my therapist ... Pres
Seph Res PPN United Against ALL Authorities	Posted - 2007.03.22 13:36:00 - [28] i fully agree with the OP but i also can understand CCP in some ways, for example: i work on a matlab project to compute several hundred EEG signals with fast fourier transform... as long as i do not try to reconfigure some parameters all works perfectly fine..then i add some code to test somethin new, e.g only another time base into computing the algorithms, the theory says it will work perfectly cause u only changed a damn time base in a written program which is set up from the dev to run proper at even such changings...and what happens? suddenly the computing time is goin up to hundrets of seconds..u sit on ur machine watchin the busy screen for 20 minutes..just to get the damn message: sorry out of memory...u change 1 value back...voila it works again... the point is that in complicated code sometimes a small change can have big effect, if its that logic or not to happen.. remembers me all time of murphys law... so CCP please if u plan somethin to fix or make improvements to the game..just tell us, we all would understand if its announced...and i even think u have some time schedules what has to be done on a day of work...so most of the time if there are no unforeseen hamster suicides the daily plan will be available at least 1 or 2 days before...i think no one says, hey lets try somethin completely different just 30 mins before he came to work...
Presidente Gallente	Posted - 2007.03.22 13:38:00 - [29] Originally by: JForce Originally by: Presidente Gallente Edited by: Presidente Gallente on 22/03/2007 13:16:48 Should I tell you something: I can understand BUT I can live with it. And you should LEARN to. EVE is not the center of my reallife. I pay EUR 10,00 per months which means: 33 euro cent a day. I am working with hardware since 20 years. I know all the simple to complex problems I have with a single PC. So I understand that CCP is dealing with state of the art concepts to make EVE happen. From this POV I see no reason to have the right to demand something because I pay simple EUR 10,00 per month. Let them do their job, be patient ... CCP will not leave us alone. And: an unexpected downtime is a great advantage to let me cool down and concentrate on RL stuff. OVER! OUT! Pres However if you had a dozen POSs to refuel and defend and manufacturing to deal with and things like that, you might expect to come on and play. And if you weren't able to, to be told in advance when they make changes that could go wrong. Yeah ... that's the dark side of MMORPG's ... they rule and frak up your RL when you want to do big biz and be a big one in those games. At this point: you should learn to live with it. Even CCP can't help you really out. They have to deal with their own unexpected problems. See it that pragmatic. Pres
B0rn2KiLL DEATHFUNK	Posted - 2007.03.22 13:40:00 - [30] Originally by: Cadela Fria Yes well, just about everyone in EVE is a Systems admin, game developer, server manager and/or work with the same things CCP do, and somehow everyone always knows better then CCP. Atleast thats the same argument everytime - "Hi I'm joe schmoe and I'm computer smart in one field or another, so therefor I unquestionably know what I'm talking about - This means you can't tell me I'm wrong...that being said, CCP is all wrong about how they do things and should bow their heads in shame and follow my every word! BOW DOWN!" you are a troll, burn slowly please. as to OP, i agree. --- Originally by: Oveur It's important to understand that EVE is a "PvP" focused game



Pages: [1] 2 :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.