| Pages: [1] 2 3 :: one page |
|
|
| Author |
Thread Statistics | Show CCP posts - 10 post(s) |
|
CCP Falcon
12284
   |
Posted - 2015.08.07 14:49:14 -
[1] - Quote
Sometimes in the world of Internet Spaceships, things don't always go according to plan.
More often than not it's because our wonderful players do something completely unexpected with content we add to the game, but occasionally there's hardware or software issues, or sometimes a combination of both, that result in extended downtime for Tranquility.
On July 25th, one of these instances occurred that resulted in the longest downtime we've had in almost half a decade...
The community requested a little more information about what exactly goes on behind the scenes when something like this happens, and of course for more technical information regarding what went wrong.
So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"
What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code...
Take a look at this blog for further information...
Enjoy!
CCP Falcon || EVE Universe Community Manager || @CCP_Falcon
Happy Birthday To FAWLTY7! <3
|
|
Aladar Dangerface
13.
Enigma Project
202
|
Posted - 2015.08.07 14:55:09 -
[2] - Quote
CCP Falcon wrote:Sometimes in the world of Internet Spaceships, things don't always go according to plan. More often than not it's because our wonderful players do something completely unexpected with content we add to the game, but occasionally there's hardware or software issues, or sometimes a combination of both, that result in extended downtime for Tranquility. On July 25th, one of these instances occurred that resulted in the longest downtime we've had in almost half a decade... The community requested a little more information about what exactly goes on behind the scenes when something like this happens, and of course for more technical information regarding what went wrong. So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!" What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code... Take a look at this blog for further information...Enjoy!
Thanks for this, and the free SP \o/
I don't need twitter.
I'm already following you.
|
Entity
X-Factor Industries
Synthetic Existence
864
|
Posted - 2015.08.07 15:11:20 -
[3] - Quote
CCP Falcon wrote:So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!
you know sometimes... things... happen when you lock 2 guys into a room...
Gòª......Gòæ...GòöGòù.Gòæ.Gòæ.GòöGòù.GòªGòæ.GòöGòùGòöGòªGòùGòöGòù
Gòæ.GòöGòùGòöGòùGòöGòú.GòöGòùGòá..Gòá GòáGòùGòáGò¥.GòæGòá GòáGò¥GòæGòæGòæGòÜGòù
Gò¬GòÉGòÜGò¥Gòæ.GòÜGò¥.GòÜGò¥Gòæ..GòÜGò¥GòæGòæGòÜGò¥.Gò¬GòÜGò¥GòÜGò¥Gòæ.GòæGòÜGò¥
Got Item?
|
Vic Vorlon
Aideron Robotics
31
|
Posted - 2015.08.07 15:16:04 -
[4] - Quote
Coincidentally, the same kind of thing happened at my work. Our logging system was running very slowly over a network, but nice and fast on our local PCs. It turned out that opening/writing/closing the log file was a lot slower than just keeping the log file open between writes. I'm sure that's a very different thing to what CCP is facing, but its nice to know we all face similar issues sometimes. |
|
CCP Goliath
C C P
C C P Alliance
2695
|
Posted - 2015.08.07 15:20:10 -
[5] - Quote
Joke's on them, I stopped watching Game of Thrones 3 eps into the last series.
CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
|
|
Chainsaw Plankton
IDLE GUNS
IDLE EMPIRE
1372
|
Posted - 2015.08.07 15:26:16 -
[6] - Quote
Sounds like the logs hit the bottle harder than a FanFest attendee
@ChainsawPlankto
|
|
CCP Goliath
C C P
C C P Alliance
2695
|
Posted - 2015.08.07 15:29:29 -
[7] - Quote
Chainsaw Plankton wrote:Sounds like the logs hit the bottle harder than a FanFest attendee
Yet still not as fast as me immediately after we opened TQ back up on the day.
CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
|
|
|
CCP Masterplan
C C P
C C P Alliance
1720
|
Posted - 2015.08.07 15:30:57 -
[8] - Quote
CCP Falcon wrote:So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"
I hope you realise what you've set in motion here...
"This one time, on patch day..."
@ccp_masterplan | Team Five-0: Rewriting the law
|
|
Hendrink Collie
Sebiestor Tribe
Minmatar Republic
51
|
Posted - 2015.08.07 15:34:34 -
[9] - Quote
CCP Masterplan wrote:CCP Falcon wrote:So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!" I hope you realise what you've set in motion here...
So... who do we lock in the closet to get citadels done in two weeks?  |
flakeonium
The Cruciform
0
|
Posted - 2015.08.07 15:40:28 -
[10] - Quote
oh the cliffhanger. 
i need the conclusion ! |
|
|
Marc Callan
Lucifer's Hammer
A Band Apart.
499
|
Posted - 2015.08.07 15:44:34 -
[11] - Quote
Illuminating. But worryingly, I got the distinct impression that CCP figured out what was causing the problem but not why - unless the underlying cause of the logging issues has since been determined?
"We are what we pretend to be, so we must be careful about what we pretend to be."
- Kurt Vonnegurt
|
drunklies
Nocturnal Romance
Cynosural Field Theory.
5
|
Posted - 2015.08.07 15:46:43 -
[12] - Quote
A blog full of words I technically understand, but have no real comprehension of what they mean.
10/10 tech blog |
Oxide Ammar
209
|
Posted - 2015.08.07 15:54:00 -
[13] - Quote
We are starting to make Blogs about that ? What a sad year for EVE development this year 
Lady Areola Fappington: -áSolo PVP isn't dead!-á You just need to make sure you have your booster, remote rep, cyno, and emergency Falcon alts logged in and ready before you do any solo PVPing.
|
Lothian Scykle
Repo.
0
|
Posted - 2015.08.07 15:59:41 -
[14] - Quote
This is (to me) a fascinating and refreshingly honest explanation of the problem. Thanks for taking the time to put this together, and for respecting your players enough to give us a peek behind the curtain.
o7o7 |
Agent Unknown
Night Theifs
incendia equus
52
|
Posted - 2015.08.07 16:01:52 -
[15] - Quote
Ah, software development. It seems to easy to reproduce a problem, but finding the root cause is the fun (and sometimes infuriating) part.
So in the case, the logs, instead of showing nothingGäó, stabbed random nodes to death? Neat. Maybe it has to do with the *way* it's reporting the logs. I know nothing about the underlying code, but a difference between the generic_logger.warn() and campaign_logger.warn() routines could be something.
It makes me wonder if the Eve problems with Windows 10 in relation to its own logging (root cause was a permissions issue, go figure) has some parallels with the way the server handles things. |
tasman devil
HUN Corp.
HUN Reloaded
53
|
Posted - 2015.08.07 16:02:40 -
[16] - Quote
CCP Falcon wrote:So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"
What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code...
I for one welcome our new CANINE overlord! :-)
PS: next time you lock two boys together, throw in a sausage and a knife 
I don't belive in reincarnation
I've never believed in it in my previous lives either...
|
Vincent Athena
V.I.C.E.
3578
|
Posted - 2015.08.07 16:03:07 -
[17] - Quote
I'm guessing this blog was somewhat delayed because you were hoping to find out why logging was being an issue before releasing the blog, so we would get the entire story. But, you still don't know?
Know a Frozen fan? Check this out
Frozen fanfiction
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
14728
|
Posted - 2015.08.07 16:04:55 -
[18] - Quote
The binary tale
GÿàGÿàGÿà Secure 3rd party service GÿàGÿàGÿà
Visit my in-game channel 'Holy Veldspar'
Twitter @Chribba
|
|
drunklies
Nocturnal Romance
Cynosural Field Theory.
6
|
Posted - 2015.08.07 16:05:04 -
[19] - Quote
Oxide Ammar wrote:We are starting to make Blogs about that ? What a sad year for EVE development this year
A large portion of the community was asking for such a blog the minute the problem was resolved.
Insert line here [This did not take development time away from Eve] |
GM Grave
Game Masters
C C P Alliance
55
|
Posted - 2015.08.07 16:05:57 -
[20] - Quote
The logs are dead, long live the logs!
Lead GM Grave | Senior Project Lead | CCP Customer Support | EVE Online | DUST 514
|
|
|
Oxide Ammar
209
|
Posted - 2015.08.07 16:16:08 -
[21] - Quote
drunklies wrote:Oxide Ammar wrote:We are starting to make Blogs about that ? What a sad year for EVE development this year A large portion of the community was asking for such a blog the minute the problem was resolved. Insert line here [This did not take development time away from Eve]
You missed the point ...totally....
Lady Areola Fappington: -áSolo PVP isn't dead!-á You just need to make sure you have your booster, remote rep, cyno, and emergency Falcon alts logged in and ready before you do any solo PVPing.
|
Plato Idari
Blue Republic
RvB - BLUE Republic
25
|
Posted - 2015.08.07 16:21:36 -
[22] - Quote
I look forward to more installments of this tale! |
Ezekiel Marr
The Real OC
ROC.
15
|
Posted - 2015.08.07 16:30:33 -
[23] - Quote
So... is castello.is a pizza place of choice for CCP? |
|
CCP Goliath
C C P
C C P Alliance
2696
|
Posted - 2015.08.07 16:41:22 -
[24] - Quote
Ezekiel Marr wrote:So... is castello.is a pizza place of choice for CCP?
We usually get pizza from Castellos yeah. It doesn't actually deliver to our area unless it's for us :p
CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
|
|
Bienator II
madmen of the skies
3354
|
Posted - 2015.08.07 17:14:32 -
[25] - Quote
its not the first time that logging causes problems in distributed software projects ;)
another classic pitfall would be when you have a highly concurrent program on a single node and as soon you enable logging suddenly nothing runs concurrently anymore
how to fix eve: 1) remove ECM 2) rename dampeners to ECM 3) add new anti-drone ewar for caldari 4) give offgrid boosters ongrid combat value
|
Projak Dynamo
Pro Synergy
49
|
Posted - 2015.08.07 17:14:45 -
[26] - Quote
While I followed about one word in twelve of that, I just wanted to say thank you to all you mad, beer swilling, pizza eating Devs and Tech-heads for getting EVE back as soon as you possibly could.
Sometimes it really cannot be solved by just turning it off and turning it back on again!
The Pro Synergy Pilot is not just a fighting man, he is a salvage expert.-áIf it is lost, in the blackness of space, he will find it. If it has been destroyed, he will loot and salvage it. If it is in his way, he will move it. If he is lucky he will be podded 20 jumps from home, for this is the closest he come to being hero.
|
Vincent Athena
V.I.C.E.
3578
|
Posted - 2015.08.07 17:38:14 -
[27] - Quote
Legacy code?
I'll make a prediction. The channel you used for campaign logging was used in the past for doing something else. You thought that code was removed, but some part of it still remains. When you started campaign logging, some old code woke up, tried to do something related to that channel, and "bad things" resulted.
Know a Frozen fan? Check this out
Frozen fanfiction
|
Arla Sarain
580
|
Posted - 2015.08.07 17:44:51 -
[28] - Quote
So, TL;DR?
FozziSov Literally did kill EVE? |
Dersen Lowery
Scanners Live in Vain
1721
|
Posted - 2015.08.07 17:59:16 -
[29] - Quote
Vincent Athena wrote:Legacy code?
I'll make a prediction. The channel you used for campaign logging was used in the past for doing something else. You thought that code was removed, but some part of it still remains. When you started campaign logging, some old code woke up, tried to do something related to that channel, and "bad things" resulted.
"I'm not a logger. I'm a pirate! Q'apla!"
Proud founder and member of the Belligerent Desirables.
I voted in CSM X!
|
Archetype 66
Association of Commonwealth Enterprises
Nulli Secunda
192
|
Posted - 2015.08.07 18:00:35 -
[30] - Quote
Thank you for that blog. So still running TQ without campains logs ? |
|
|
|
|
| |
|
| Pages: [1] 2 3 :: one page |
| First page | Previous page | Next page | Last page |