| Pages: 1 2 3 4 5 6 7 :: [one page] |
| Author |
Thread Statistics | Show CCP posts - 23 post(s) |
|
CCP Wrangler
   |
Posted - 2007.09.15 19:49:00 -
[1]
This Monday it was brought to our attention that an alliance couldn't get Sovereignty 4 in their capital system even though they should. Looking into it we concluded that their capital station was a conquerable station which shouldn't be possible to set as a capital station. Fixing this required changing procedures in the database so attempting to set a conquerable station as capital station causes an error.
Unfortunately, due to unforeseen behavior in the server code a certain argument was passed to the procedure, which did not have anticipated default value, causing the procedure to error on every call from conquerable stations that were set as capital stations, of which there are 10 of on Tranquility.
While hotfixing this we failed to notice that the error flag code we copied was bad, that error condition can possibly have caused other crashes, in the sense that it did not close the transaction. This in turn led to that every call to this procedure would lock up a session to the database as it waited indefinitely for the transaction to be either rolled back or closed. The sessions locking up created a plethora of different locks all over the database, causing us to chase for a ghost of the problem. Eventually this led to a database fail-over which meant Tranquility crashed.
This oversight has been located in two other procedures as well, but none of them are called often enough on Tranquility to cause any real problems.
To sum it up, a series of unfortunate exceptional cases set up an environment on TQ wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up. As a result of this, our hotfixing protocols are being revised to prevent this from happening again in the future.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
|
CCP Atropos
|
Posted - 2007.09.15 19:51:00 -
[2]
\o/
|
|
Larg Kellein
Caldari
Agony Unleashed
Agony Empire
|
Posted - 2007.09.15 19:52:00 -
[3]
Woah... POS wars really are killing the game, who would have thought?
Nice catch, CCP dudes and dudettes.
 Originally by: Roy Batty68
My software has wronged me!!! And it's immediately "sorry". Well, ok then. I suppose I'm not so upset at my software if it appologizes.
|
|
CCP Prism X
|
Posted - 2007.09.15 19:53:00 -
[4]
Working on Saturdays is so much more giving when it results in something. 
~ Prism X
EvE Lead Database Developer,
Relocating your character to a cozy, giant secure container since 2006. |
|
Blackback Starkiel
|
Posted - 2007.09.15 19:56:00 -
[5]
Sounds like a very difficult bug to catch. Let's hope that does it then!
.
Griatch
Holographic gallery of my artwork |
|
CCP Mindstar
|
Posted - 2007.09.15 19:57:00 -
[6]
|
|
Shiner BockBeer
Go Go Gadget ForumPostingAlt
|
Posted - 2007.09.15 19:58:00 -
[7]
Excellent work. Thanks for the update. :)
|
Deathdealerdoc
|
Posted - 2007.09.15 20:00:00 -
[8]
good job!!!!
|
Onderzoeker 2
|
Posted - 2007.09.15 20:01:00 -
[9]
good job CCP 
|
TigerWoman
|
Posted - 2007.09.15 20:03:00 -
[10]
superb work!
i owe you a beer on the fanfest!
|
Pirate Tom
|
Posted - 2007.09.15 20:05:00 -
[11]
Originally by: "CCP Wranglermuffin"
...wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up...
So in a roundabout sort of way, it really was BoB and Goonfleet (et al) causing the crash! |
Lance Fighter
Amarr
|
Posted - 2007.09.15 20:08:00 -
[12]
*tinfoil-hattery* Bob is killing the servers!!!! noooO! nerf bob!
*/tinfoil-hattery*
Great job CCP, kudos
|
torswin
|
Posted - 2007.09.15 20:11:00 -
[13]
http://www.yonkis.com/mediaflash/1000000.swf
thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
---
Signature radius: 150 mm
Unless explicitly stated, this post does not represent my alliance, corporation, my own, or any other living organism's view. |
Price Watcher
|
Posted - 2007.09.15 20:11:00 -
[14]
Well done.
POST WITH YOUR ALT!
The Shame o' The Galaxy |
|
CCP Wrangler
|
Posted - 2007.09.15 20:11:00 -
[15]
Originally by: Pirate Tom
Originally by: "CCP Wranglermuffin"
...wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up...
So in a roundabout sort of way, it really was BoB and Goonfleet (et al) causing the crash!
No, they were not involved. I'd also like to clarify that this problem was not caused by any Alliance, we messed up and not any player(s).
Originally by: torswin
http://www.yonkis.com/mediaflash/1000000.swf
thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
You people will never let me forget that, will you?  
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
|
CCP John Proctor
|
Posted - 2007.09.15 20:12:00 -
[16]
I can go sleep now 
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
|
|
Pirate Tom
|
Posted - 2007.09.15 20:16:00 -
[17]
Wrangler I was going for the obvious poke at the 'OMG BOB' and 'OMG GOONS' crowd. I know it was a code error. You explained that pretty clearly. |
Claudia Jefferson
|
Posted - 2007.09.15 20:18:00 -
[18]
Thats the best news all week ;)
|
|
CCP Wrangler
|
Posted - 2007.09.15 20:19:00 -
[19]
Originally by: Pirate Tom
Wrangler I was going for the obvious poke at the 'OMG BOB' and 'OMG GOONS' crowd. I know it was a code error. You explained that pretty clearly.
I figured, I just wanted to make sure people understood that it was no Alliances fault that this happened.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
torswin
|
Posted - 2007.09.15 20:20:00 -
[20]
Originally by: CCP Wrangler
Originally by: torswin
http://www.yonkis.com/mediaflash/1000000.swf
thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
You people will never let me forget that, will you?
nevar forget!
Takk fyrir, CCP!
---
Signature radius: 150 mm
Unless explicitly stated, this post does not represent my alliance, corporation, my own, or any other living organism's view. |
Elmicker
The Phoenix Rising
FreeFall Securities
|
Posted - 2007.09.15 20:20:00 -
[21]
♥
|
Peter VonThal
|
Posted - 2007.09.15 20:21:00 -
[22]
Nice! I can imagine it wasn't fun trying to hunt something like that down.
|
Drykor
Minmatar
Jericho Fraction
The Star Fraction
|
Posted - 2007.09.15 20:28:00 -
[23]
good job
|
Sylia
Minmatar
Omerta Syndicate
Exuro Mortis
|
Posted - 2007.09.15 20:31:00 -
[24]
Guess sharkbait is sleeping after a few sleepless nigths now eh :)
Grats CCP on getting it fixed though, we players appreciate it
If I can save some iskies I'll buy a sexy dev or 2 a beer at fanfest
|
Neroo Tal
Minmatar
Ambart Valaina
|
Posted - 2007.09.15 20:32:00 -
[25]
'grats on solving the problem CCP! Well done! Now, all of you go take a nice long deserved nap!
Ambart Valaina -= Becomes sometimes Fate is truly Divine!
Ambart Valaina -= Becomes sometimes Fate is truly Divine!
-=Freedom over all else, Honour over Freedom
-=The pain of loss cannot compare to the pain of captivity. |
Elseer Radak
Gallente
Paxton Industries
Paxton Federation
|
Posted - 2007.09.15 20:33:00 -
[26]
well caught, sentients, well caught! such a subtle error, such awful damage!
\o/ Extra Vitoc for the hamsters!!! heck, extra Vitoc for Dev, QA and OPS \o/
now go home and get some sleep you all clearly deserve it!
Elseer R.
--
Reality: A consensual experience
|
Reem Fairchild
|
Posted - 2007.09.15 20:34:00 -
[27]
Good job, CCP!!!
|
|
CCP Gangleri
|
Posted - 2007.09.15 20:34:00 -
[28]
Originally by: Sylia
Guess sharkbait is sleeping after a few sleepless nigths now eh :)
Grats CCP on getting it fixed though, we players appreciate it
If I can save some iskies I'll buy a sexy dev or 2 a beer at fanfest
All devs are sexy, you might want to start saving up.
|
|
Ex0101
Malicious Intentions
Cult of War
|
Posted - 2007.09.15 20:34:00 -
[29]
\o/ Nice work guys, thanks for the explanation
--
|
ViolenTUK
Gallente
Vindicated Exiles
|
Posted - 2007.09.15 20:36:00 -
[30]
In this crash the entire "3asy company" Alliance was lost of the database. All of the alliance members were kicked from the alliance and 3asy company doesnt show in the rankings list. Please CP give us back our alliance. 
www.eve-players.com |
Telkkar
|
Posted - 2007.09.15 20:40:00 -
[31]
Originally by: CCP John Proctor
I can go sleep now
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
Oooh... sorry to hear that man :(
Thanks for finding the problem CCP, and thank you :D
|
Taloth Saldono
|
Posted - 2007.09.15 20:41:00 -
[32]
Definitely worth a 'Nice Work, Guys'.
I've done enough troubleshooting/debugging myself to know what a tedious task it can be to locate the ACTUAL cause (And saying 'O... of course', after finding it), but fortunately the satisfaction after solving such issues is of equal, if not higher, magnitude.
Again, thank you for the tremendous effort in minimizing the stress on the player-base by monitoring the system to prevent as much crashes as humanly possible.
Please pass on the ty to the external support guys for their effort in resolving this nuisance.
|
Joey Judas
Gallente
Ramraiders
|
Posted - 2007.09.15 20:45:00 -
[33]
Well done CCP, this type of news it what is severly lacking in other competing (and inferior) MMO's
|
Seriya
Caldari
RSP Enterprises
|
Posted - 2007.09.15 20:46:00 -
[34]
Good show chaps!
|
Pistonbroke
D00M.
Triumvirate.
|
Posted - 2007.09.15 20:49:00 -
[35]
Edited by: Pistonbroke on 15/09/2007 20:55:15
Clarification please.... does this mean that conquerables can not ever be set as capital station?
|
Brunswick2
coracao ardente
Triumvirate.
|
Posted - 2007.09.15 20:50:00 -
[36]
our bad
|
Larieen
|
Posted - 2007.09.15 20:50:00 -
[37]
Yeah good job guys,but as it stands i have been with out eve for 7 days and every day i looked for the 36800 patch and its not available.Would really like to update my game so i can play with out the Incompatible (Protocol) message!It is nowhere on this site!
|
Kalissa
D00M.
Triumvirate.
|
Posted - 2007.09.15 20:51:00 -
[38]
Does this mean if you've set a conquerable station to be a capital already but it's not kicked in yet (you've not done the 30 days) that it wont kick in when the 30 days are up?
|
|
Kaemonn
Forum Moderator
|
Posted - 2007.09.15 20:51:00 -
[39]
\o/
forum rules | [email protected]| Eve-CCG
Originally by: kieron: off duty
You dont have to swallow!
|
|
Khamal Jolstien
Caldari
Lucky Hydra Corp
SMASH Alliance
|
Posted - 2007.09.15 20:51:00 -
[40]
Wow. I've never known an MMO company to have such transparency on these issues. Usually you just hear "Database was fixed". Good job CCP.
|
|
Rauth Kivaro
Forum Moderator
|
Posted - 2007.09.15 20:53:00 -
[41]
Great stuff! Good job to everyone on the team who put in the hours to track this down.
Oh, and next time, get some sleep, CCP John Proctor. ;)
forum rules | [email protected] | Our Website
|
|
Sirex007
|
Posted - 2007.09.15 20:55:00 -
[42]
Edited by: Sirex007 on 15/09/2007 20:54:42
see, no biggie. :-)
- really though, great bug squish. hopefully the people whining will stfu now.
|
ArchenTheGreat
Caldari
D00M.
Triumvirate.
|
Posted - 2007.09.15 20:56:00 -
[43]
Good job guys. I know how hard it's to work under such pressure.
Originally by: CCP Wrangler
This in turn led to that every call to this procedure would lock up a session to the database as it waited indefinitely for the transaction to be either rolled back or closed.
You should really use some language which prevents this kind of thing.
|
|
Eldo Davip
Forum Moderator
|
Posted - 2007.09.15 20:58:00 -
[44]
Get some sleep guys. Well done.
___
|
|
|
IC Tsuki Kiyuu
Caldari
ISD Interstellar Correspondents
|
Posted - 2007.09.15 21:03:00 -
[45]
I'd been thinking all along that my screwy html tags in the latest IC article were to blame. I'd been having these nightmares that I'd slowly set off a chain reaction that would eventuate in the deletion of Jita and the unlocking of the gate back to Earth, out of which poured a vast army of epileptic Jovian furriers that enslaved us all.
Imagine my relief. Turns out we are safe from those bloodthirsty monsters for another day! \o/
Good work on delivering so quickly on the issue, that hint in the news article was right!
|
|
Zero shadow1
|
Posted - 2007.09.15 21:05:00 -
[46]
The server was down?
|
Aleyah Dawnborn
Caldari
SMASH Alliance
|
Posted - 2007.09.15 21:07:00 -
[47]
Well done indeed, and thanks for keeping us up-to-date through the entire mess.
Three cheers for CCP, and the guys-that-fix-stuff!
---
My God! It's logic! Flee! |
Bojan Z
|
Posted - 2007.09.15 21:08:00 -
[48]
Nice work CCP!
And many thanks for all the explanations on the problem, seems like it was not as easy to fix as the whiner thought first.
Enjoy your WE and your beers now; these are well deserved. \o/
|
|
CCP Wrangler
|
Posted - 2007.09.15 21:08:00 -
[49]
Originally by: ViolenTUK
Edited by: ViolenTUK on 15/09/2007 20:47:59
In this crash the entire "3asy company" Alliance was lost of the database. All of the alliance members were kicked from the alliance and 3asy company doesnt show in the rankings list. Please CCP give us back our alliance.
I'm sorry to hear that, can your CEO create a petition on it, or someone else within that alliance? I'll send off some emails too so we can get it handled.
Originally by: Pistonbroke
Edited by: Pistonbroke on 15/09/2007 20:55:15
Clarification please.... does this mean that conquerables can not ever be set as capital station?
I believe that was never the intention.
Originally by: Kalissa
Does this mean if you've set a conquerable station to be a capital already but it's not kicked in yet (you've not done the 30 days) that it wont kick in when the 30 days are up?
That is something I'll have to check into.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Barwinius
Ars ex Discordia
|
Posted - 2007.09.15 21:09:00 -
[50]
Sounds like a nightmare... Nice work getting it resolved! |
Resla
|
Posted - 2007.09.15 21:12:00 -
[51]
Awesome! One more reason to check more accuratly petitions
|
GEAL
|
Posted - 2007.09.15 21:13:00 -
[52]
good work ccp
|
adriaans
Amarr
Advanced Capital Ship Designs
|
Posted - 2007.09.15 21:15:00 -
[53]
Good work!
must have been a fun hunt after the error
--sig--
Knowledge is power! |
Kalissa
D00M.
Triumvirate.
|
Posted - 2007.09.15 21:16:00 -
[54]
It would be good if you could check into that Wrangler, a hell of a lot of planning on our side has gone into Sov 4 and this could foul a lot of it up.
|
exxxie
Debiloff's Vanguard
The Makhai
|
Posted - 2007.09.15 21:16:00 -
[55]
Good work, well done Dev-types.
--
Originally by: CCP Sharkbait
you people make me question the sanity of the human race sometimes 
|
ForceM
Gallente
POS Builder Inc.
|
Posted - 2007.09.15 21:17:00 -
[56]
Gratz on finding the solution.
I aint a programmer .. and im not gonna say i completely understand what you just said.
Bu im jsut happy at least the Title of the topic which says enough for me :)
-----
Quote:
Dun kill me eris
coming to fanfest by any chance? eris
*GULP*
|
ViolenTUK
Gallente
Vindicated Exiles
|
Posted - 2007.09.15 21:19:00 -
[57]
Edited by: ViolenTUK on 15/09/2007 21:19:48
A petition has been sent for this issue. I hope the 3asy alliance is back soon.
3asy Company
www.eve-players.com |
Disteeler
Neh'bu Kau Beh'Hude
Ushra'Khan
|
Posted - 2007.09.15 21:20:00 -
[58]
\o/
PD: I love the level of detail you go when explaining insiders of your server infrastructure. Lots of us are technology lovers so we appreciate it a lot, and we learn how those mmo are made ^^
|
Andrue
Amarr
|
Posted - 2007.09.15 21:26:00 -
[59]
Originally by: Larieen
Yeah good job guys,but as it stands i have been with out eve for 7 days and every day i looked for the 36800 patch and its not available.Would really like to update my game so i can play with out the Incompatible (Protocol) message!It is nowhere on this site!
7 days? Wouldn't it have been quicker to just download the game from scratch?
--
(Battle hardened industrialist)
[Brackley, UK]
My budgie can say "ploppy bottom". You have been warned. |
glennkari
Gallente
Lions Club
|
Posted - 2007.09.15 21:31:00 -
[60]
Thank you CCP!
Thank you for a great game, and the hard work and long hours you guys poor into it 
|
Gabriel Tenlegall
Gallente
Kharak Confederation
|
Posted - 2007.09.15 21:33:00 -
[61]
You all deserve a beer round for free. I will remember it when I travel to Iceland someday. Many thanks for your fix and your transparency CCP. Good work!
|
Dimning
|
Posted - 2007.09.15 21:38:00 -
[62]
I postulate to exemplary ban the characters who actually did turn conquerable stations into capital stations even though they knew well that it is not how this should work. This is a typical exploitation of unintended game features which is bannable.
Thank you.
|
|
CCP Explorer
|
Posted - 2007.09.15 21:39:00 -
[63]
Originally by: Larieen
Yeah good job guys,but as it stands i have been with out eve for 7 days and every day i looked for the 36800 patch and its not available.Would really like to update my game so i can play with out the Incompatible (Protocol) message!It is nowhere on this site!
The patches are here.
Erlendur S. Thorsteinsson
Software Director
EVE Online, CCP Games |
|
Aaron Mirrorsaver
The Coalition Of Buccaneers
Mercenary Services
|
Posted - 2007.09.15 21:42:00 -
[64]
could this have caused issues as well?
--
Greater Love hath no man than this; that a man lay down his life for his friends.
Mercenary Services Recruiting |
Marine HK4861
Caldari
Seoltachd
|
Posted - 2007.09.15 21:45:00 -
[65]
Originally by: CCP Explorer
Originally by: Larieen
Yeah good job guys,but as it stands i have been with out eve for 7 days and every day i looked for the 36800 patch and its not available.Would really like to update my game so i can play with out the Incompatible (Protocol) message!It is nowhere on this site!
The patches are here.
More specifically:
http://ccp.vo.llnwd.net/o2/evepatch36191-36800.exe
|
Verone
|
Posted - 2007.09.15 21:45:00 -
[66]
Originally by: CCP Wrangler (translated into Veroneish)
This Monday RABBLERABBLE RABBLE RABBLE RABBLE Sovereignty 4 in their capital system even though RABBLERABBLERABBLE RABBLE RABBLE RABBLE RABBLE conquerable station RABBLE RABBLE RABBLE station. Fixing this required RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE an error.
Unfortunately, due to unforeseen behavior in RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE Tranquility.
While hotfixing this we failed to notice that the RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE transaction. This in turn led to RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLERABBLE RABBLERABBLE RABBLE either rolled back or closed. The sessions locking up created a RABBLE RABBLE RABBLE database, causing us to chase for a RABBLE RABBLE of the problem. Eventually this led to a RABBLE fail-over which meant Tranquility crashed.
This oversight has been located in RABBLE as well, but none of them are RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE RABBLE real problems.
To sum it up, RABBLE RABBLE RABBLE RABBLE RABBLERABBLE RABBLE RABBLE RABBLERABBLE RABBLERABBLE RABBLE RABBLERABBLE RABBLeRABBLE RABBLE RABBLE prevent this from happening again in the future.
Clear as Crystal
Tob job guys <3
>>> TRIBUTE TO A FALLEN WINGMAN <<<
|
Deadly Henkie
|
Posted - 2007.09.15 21:47:00 -
[67]
Good work guys! As a developer (I also use MSSQL) I know it can be a pain in the *ss to find bugs 
|
SFShootme
Reikoku
Band of Brothers
|
Posted - 2007.09.15 22:05:00 -
[68]
\o/
nice job fellas
-
Tho shall give Life, for Life. |
Sanfrey Statolomy
Gallente
Project Nemesis
|
Posted - 2007.09.15 22:05:00 -
[69]
Awesome! That must have been a real Eureka moment. Thanks for the technical explanation too, I'm sure lots of others work in IT like me and were reading the post with lots of "aha!!" and "oh yeah that would be a pain to find" commentary!! Now take a short holiday. There.. did you enjoy it? back to work! *Whip******
|
Kuzya Morozov
|
Posted - 2007.09.15 22:11:00 -
[70]
You have to admit, CCP is really dedicated to their customers. Kept us updated and apologized many times, tried their best to fix this, and apparently have. Never seen such dedicated customer support, bravo CCP
|
Major Stormer
Caldari
Copperhead Inc.
Molotov Coalition
|
Posted - 2007.09.15 22:12:00 -
[71]
Cheers guys!
|
Delta Hammer
|
Posted - 2007.09.15 22:16:00 -
[72]
There's nothing quite so humbling as to find out that other people were smart enough to fix a problem that I couldn't even understand the explanation of.
|
Wrayeth
Shiva
Morsus Mihi
|
Posted - 2007.09.15 22:26:00 -
[73]
WOOT! Congrats! Anyone who can find an error like that in millions of lines of code has my respect.
-Wrayeth
n00b Extraordinaire
"Look, pa! I just contributed absolutely nothing to this thread!" |
|
CCP Prism X
|
Posted - 2007.09.15 22:28:00 -
[74]
Edited by: CCP Prism X on 15/09/2007 22:28:33
Answers to some questions:
According to design conquerable stations should not be able to become capital stations. They do count towards constellation sovereignty but cannot be capitals. They wont be allowed to become capital stations in the future either. Owners of such stations should already have been contacted by GMs but what resolution will be offered is not up to me.
~ Prism X
EvE Lead Database Developer,
Relocating your character to a cozy, giant secure container since 2006. |
|
wasn't me
|
Posted - 2007.09.15 22:32:00 -
[75]
Is it remotely possible that you broke the locking down of blueprints again? I had it twice now this night that after proposing a lockdown vote, getting it voted on, that the bleuprints are unlocked again minutes after voting on it (and thus less then 10 minutes after initiating the lockdown vote) |
Swamp Ziro
GoonFleet
GoonSwarm
|
Posted - 2007.09.15 22:32:00 -
[76]
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
|
Celestia Tinkerbell
|
Posted - 2007.09.15 22:33:00 -
[77]
As a new programmer to this field, still going to school at ITT-Technical Institute in Burr Ridge, IL, I can understand how one little piece of code, or even a switch called in that code, can be a needle in a haystack when being looked for. I am sure I am underestimating the number of lines of code for a game this vast as being in the millions. That being said, Just thinking about it gives me a headache!
Good job catching the code errors, and let's hope that the next error is a mouse, and not a needle in that haystack!
|
mistole
Gallente
Swordbruden Mining and Security Service Inc.
Te-Ka
|
Posted - 2007.09.15 22:35:00 -
[78]
And you have delivered....as specified in the hint....good job...and I feel sorry for ppl that threatened to stop playing.
|
Alerion
Acquire
|
Posted - 2007.09.15 22:41:00 -
[79]
Thanks for your dedication and hard work and for solving the problem!
Most of all I appreciate the detailed information about what was wrong and how it was fixed.
Good work! Now get some sleep :-)
/Al |
Aliaa Atreides
|
Posted - 2007.09.15 22:48:00 -
[80]
Nice going guys... glad you got that fixed it was starting to get really annoying!
Thanks for the hard work and dedication
|
Murtala
|
Posted - 2007.09.15 22:48:00 -
[81]
Who found the problem, you CCP guys, the Microsoft guys, IBM people or did you get Chuck Norris.
Can you name the name of the person responsible for the code, just so we can poke him with twigs and branches for a little while.
just for fun
Good work eitherway
|
|
CCP Explorer
|
Posted - 2007.09.15 22:53:00 -
[82]
Originally by: Swamp Ziro
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
Yes, seriously.
Erlendur S. Thorsteinsson
Software Director
EVE Online, CCP Games |
|
niall08
|
Posted - 2007.09.15 22:55:00 -
[83]
Great work and thanks for putting in the long hours to fix this.
|
Gerome Doutrande
Coreli Corporation
Exuro Mortis
|
Posted - 2007.09.15 22:59:00 -
[84]
I had to deal with such "undercover transaction calls" in the past as well, and they are often annoying and difficult to locate as a problem source. gg for identifying them.
|
Mininggirlie
|
Posted - 2007.09.15 23:14:00 -
[85]
Wrangler (or someone from CCP) :
the reason we saying we are not getting our ships back , is that GMs are still answering our petitions that they can not find anything wrong on CCPs end , and it must be on our the customers end . and that the reimbursement has been turned down.
Will we be getting reimbursement for ships for this mess??
Shands
Shads
|
Jonathos
|
Posted - 2007.09.15 23:15:00 -
[86]
Great work CCP!
|
Sleepkevert
Paradox v2.0
Interstellar Alcohol Conglomerate
|
Posted - 2007.09.15 23:23:00 -
[87]
Originally by: CCP Explorer
Originally by: Swamp Ziro
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
Yes, seriously.
This shows how far some CCP employees will go for their game.
♥ you CCP
Sign my sig |
Apoc77
Federation of Synthetic Persons
YouWhat
|
Posted - 2007.09.15 23:25:00 -
[88]
Originally by: CCP Prism X
Edited by: CCP Prism X on 15/09/2007 22:28:33
Answers to some questions:
According to design conquerable stations should not be able to become capital stations. They do count towards constellation sovereignty but cannot be capitals. They wont be allowed to become capital stations in the future either. Owners of such stations should already have been contacted by GMs but what resolution will be offered is not up to me.
Hey.
everybody is happy that tranq is finally running smooth again and so am i. About the other issue, me and my alliance members are directly affected of, i'm not so contentet. Is there any special reason WHY conquerables can not be set as capitals? Im in permanent conversation with more than one GM for about a month and none could tell me why this has never been intended.
For me this sounds like an excuse and i see parallels to the time when station services on outpost were shootable but not on conquerables. Everyboy immediately saw that this are unfair conditions for all outpost owners and this certain issue has been solved quickly within days. I fear that the issue i'm talking about just does not effect that many people to raise the voice of the eve-community up enough to get this fixed too.
A frustrated player.
Apoc77
[FSP.]
P.S.: No resolution/reimbursement will be offered.
|
Spindeln
Duty.
|
Posted - 2007.09.15 23:32:00 -
[89]
Thank you, and sleep well
|
shuckstar
Gallente
Hauling hogs
Genesis Industrial Foundation
|
Posted - 2007.09.15 23:39:00 -
[90]
Originally by: CCP John Proctor
I can go sleep now
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
John you are one dedicated guy, thnx and thnx to all CCP employees for the hard work
|
|
CCP Prism X
|
Posted - 2007.09.15 23:40:00 -
[91]
True, YouWhat suffers from this mistake of ours.
However, this topic is not about that. I believe your alliance is in contact with our GMs to try and resolve the issue and I hope that resolution will be fair and square. I'm not trying to diminish the problem you have in any way at all. I just don't want this issue to become derailed. We're on your case to the extent that I've known about it for almost a week now, no worries it will be resolved one way or the other.
~ Prism X
EvE Lead Database Developer,
Relocating your character to a cozy, giant secure container since 2006. |
|
Kobadon
|
Posted - 2007.09.15 23:47:00 -
[92]
i am still not able too get on it just saying checking status i try to join it sead the sever has timed out not sure it may be me
|
Emmerlaine
Gallente
Crab and Krawdad Enterprises
|
Posted - 2007.09.15 23:52:00 -
[93]
Originally by: Apoc77
Originally by: CCP Prism X
Edited by: CCP Prism X on 15/09/2007 22:28:33
Answers to some questions:
According to design conquerable stations should not be able to become capital stations. They do count towards constellation sovereignty but cannot be capitals. They wont be allowed to become capital stations in the future either. Owners of such stations should already have been contacted by GMs but what resolution will be offered is not up to me.
Hey.
everybody is happy that tranq is finally running smooth again and so am i. About the other issue, me and my alliance members are directly affected of, i'm not so contentet. Is there any special reason WHY conquerables can not be set as capitals? Im in permanent conversation with more than one GM for about a month and none could tell me why this has never been intended.
For me this sounds like an excuse and i see parallels to the time when station services on outpost were shootable but not on conquerables. Everyboy immediately saw that this are unfair conditions for all outpost owners and this certain issue has been solved quickly within days. I fear that the issue i'm talking about just does not effect that many people to raise the voice of the eve-community up enough to get this fixed too.
A frustrated player.
Apoc77
[FSP.]
P.S.: No resolution/reimbursement will be offered.
Heh, sounds like CCP fooled 10 different alliances into putting a lot of work into trying to get sov 4. Guess the joke's on us players again. 
|
El'essar Viocragh
Minmatar
FSK23
|
Posted - 2007.09.16 00:04:00 -
[94]
Grouphug!
And on the station thing, denying conq. stations the ability to be capital is the future-proof way of making sure that at no time someone will be able to get Sov4 without having built at least a single outpost (in case there will be in some future expansion a new constellation that has 3 conq. stations - I don't think there is one right now).
--
[17:47] <Mephysto> its dead, jim |
Grainsalt
Free Corp
|
Posted - 2007.09.16 00:05:00 -
[95]
Originally by: CCP Wrangler
This Monday it was brought to our attention that an alliance couldn't get Sovereignty 4 in their capital system even though they should. Looking into it we concluded that their capital station was a conquerable station which shouldn't be possible to set as a capital station. Fixing this required changing procedures in the database so attempting to set a conquerable station as capital station causes an error.
Unfortunately, due to unforeseen behavior in the server code a certain argument was passed to the procedure, which did not have anticipated default value, causing the procedure to error on every call from conquerable stations that were set as capital stations, of which there are 10 of on Tranquility.
While hotfixing this we failed to notice that the error flag code we copied was bad, that error condition can possibly have caused other crashes, in the sense that it did not close the transaction. This in turn led to that every call to this procedure would lock up a session to the database as it waited indefinitely for the transaction to be either rolled back or closed. The sessions locking up created a plethora of different locks all over the database, causing us to chase for a ghost of the problem. Eventually this led to a database fail-over which meant Tranquility crashed.
This oversight has been located in two other procedures as well, but none of them are called often enough on Tranquility to cause any real problems.
To sum it up, a series of unfortunate exceptional cases set up an environment on TQ wherein 10 stations on the entire server could repeatedly lock sessions in the DB until it had to do a failover to free them up. As a result of this, our hotfixing protocols are being revised to prevent this from happening again in the future.
I find it strangley scarey that I understand that statement... But fair enough.. EVE is a complex beast.. And Transactions are more so..
---
http://oldforums.eveonline.com/?a=topic&threadID=554257
|
|
CCP Prism X
|
Posted - 2007.09.16 00:09:00 -
[96]
Originally by: Emmerlaine
Originally by: Apoc77
Originally by: CCP Prism X
Edited by: CCP Prism X on 15/09/2007 22:28:33
Stuff.
Stuff.
Heh, sounds like CCP fooled 10 different alliances into putting a lot of work into trying to get sov 4. Guess the joke's on us players again.
I've answered this already. Please refrain from derailing this topic. If you feel my previous answer was inadequate create your own topic.
~ Prism X
EvE Lead Database Developer,
Relocating your character to a cozy, giant secure container since 2006. |
|
Cermanius
|
Posted - 2007.09.16 01:03:00 -
[97]
CCP, my hat is off to all the people that worked on this. I know first hand how hard it is to trouble shoot issues of that type, for I've done it for a website I work for. Days spent sifting through core dumps, log files, and such.
Again, excellent work. Good to know that information flows freely at CCP. Now everyone should go rest... for 3 days...
|
Jaku Lonaru
|
Posted - 2007.09.16 01:20:00 -
[98]
I want to commend CCP for the way they handled this DB issue. Not just for the countless hours and sleepless nights they spent working on the issue, and the no doubt oodles of money spent on outside consultants. But especially for the transparency. I think this is the first time since I've been playing EVE that I felt completely informed and not the least bit in the dark about what had gone wrong and what was being done to solve the problem. I think this level of transparency is what is needed to continue to reassure the user base that CCP is in fact doing everything they can to maintain a stable and enjoyable player experience.
As software developer, I understand how difficult it can be to deliver bug free software. No amount of testing can guarantee bug free software. And testing all permutations of a system as large and complex as EVE must be a truly daunting task. With each new class of error, the testing and QA procedures are updated to reduce the chance of that type of error slipping though in the future. This is the nature of software development and CCP has done a commendable job, especially given their need to balance stability with feature growth.
It is my sincere hope that CCP will continue to be as forthcoming and transparent with future issues as they have been with this issue.
Admitting to and being open about mistakes is not a sign of weakness, but a sigh of strength.
Keep up the good work CCP.
|
Ilfar
Minmatar
Manta Ice
Ilfar Collective Enterprises
|
Posted - 2007.09.16 01:49:00 -
[99]
Big cheers for the explanation, very reassuring to get told what went wrong rather than just "x broke, y was down as a result". :3
*-----------------------*
Press to test...
*click*
Release to detonate... |
Great Artista
Caldari
Space Perverts and Forum Warriors United
Cult of War
|
Posted - 2007.09.16 02:20:00 -
[100]
Excellent work CCP! \o/
_______
◕◡◕
Space perverts and forum warriors united.
[PERVS]
My opinions rather rarely represent my corp, not to mention my alliance hihihi... |
Cagot
Gallente
Arcanus
|
Posted - 2007.09.16 02:34:00 -
[101]
Excellent debugging job, team, and thanks for the detailed explanation. Sounds like this can't have been an easy one to track down.
Woot! \o/
|
flakeonium
Amarr
The Cruciform
|
Posted - 2007.09.16 02:44:00 -
[102]
horror week for devs i guess yes, but could someone tell CCP John Proctor he does not have to risk his health for me ?
(i assume he is alright now)
--------------------------------------------------
*<(:-} |
Reachok
Amarr
Omiracon Technologies
|
Posted - 2007.09.16 02:54:00 -
[103]
Good server detective work!!
|
Tkar vonBiggendorf
Gallente
Snake Eyes Inc
deadspace society
|
Posted - 2007.09.16 03:04:00 -
[104]
Originally by: CCP John Proctor
I can go sleep now
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
CCP John Proctor, I salute you. Now, don't do that again.
Thanks CCP. I'm sure ya'll are feeling mighty relieved about now.
|
Jack Randall
Caldari
AFK
|
Posted - 2007.09.16 03:11:00 -
[105]
Congrats for hunting down and solving this bug guys!
Btw I love this one:
"The sessions locking up created a plethora of different locks all over the database, causing us to chase for a ghost of the problem"
Do you mind if I use it in my work?
Goddam Scotty! Don't have all day! |
AutoDJ
|
Posted - 2007.09.16 04:55:00 -
[106]
honestly i dont think concurable stations should be allowed to be set to capitals .
if an alliance wants a capital i think they should have to have an outpost either in the same system or one else where that would have avoided the the ghost bug as well
any way good job on the work
this is an alt btw
|
Victoria Ehr
Twisted Attitude
Executive Outcomes
|
Posted - 2007.09.16 05:02:00 -
[107]
Edited by: Victoria Ehr on 16/09/2007 05:03:46
Thanks CCP for all your efforts with this unique game and fixing that aweful bug, once again the support guys save the fleet!
Demand answers and inherit lies; Ask questions and inherit truth. Ramileus |
shortspecialbus
GoonFleet
GoonSwarm
|
Posted - 2007.09.16 05:56:00 -
[108]
Thanks for getting this fixed, it's been a frustrating week for all. Good work guys.
|
Dred'Pirate Jesus
Amarr
Ministry of War
|
Posted - 2007.09.16 06:36:00 -
[109]
Yay!
Originally by: David Hackworth
ň If you find yourself in a fair fight, you didn't plan your mission properly.
|
Martineth
|
Posted - 2007.09.16 06:42:00 -
[110]
Good work. and i DO NOT MEAN RECTIFYING THIS.
It is good that you kept us posted, made us take part in a way. This is the first time i felt that the Dev Team is beside me. This is the CCP we are hoping for. Thumbs up. You ever strand in Greece leme know. I buy the Coffe. (Beer here sucks big time but you can allways try OUZO power.)
|
Mad Pedro
|
Posted - 2007.09.16 06:54:00 -
[111]
Is the server still down, or is it just me?
|
Althea Nar'agh
StarHunt
Fallout Project
|
Posted - 2007.09.16 07:14:00 -
[112]
\0/
Nice one guys! I knew it's players fault 
Now all of you go have some good sleep!!!
Onece more gratz on finding it :)
--------------------------
War. War never changes
EvE Training Monitor / iMonitor / iCEO
EvE iHelper |
Msquare
|
Posted - 2007.09.16 07:22:00 -
[113]
Originally by: CCP John Proctor
I can go sleep now
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
As an IT troubleshooter - Been there, done it, ... almost drove off the road into the ditch on the way back to the hotel. (Oh this was years ago, when I was a young man, full og bragging and folly.)
Anyhow, all is well that ends well, and there is that "kick" when you have finally nailed the sucker!
I also note, for all of those (negative) posts about the database - it wasn't the database, it was an application issue. (And we could debate at length whether the DB should have been able to show no.of.open.transactions, lost.session.count, or whatever metric was needed to point the bughunters in the right direction; but we wont.)
Lets play ! |
Andrue
Amarr
|
Posted - 2007.09.16 07:32:00 -
[114]
Originally by: Sanfrey Statolomy
Awesome! That must have been a real Eureka moment. Thanks for the technical explanation too, I'm sure lots of others work in IT like me and were reading the post with lots of "aha!!" and "oh yeah that would be a pain to find" commentary!! Now take a short holiday. There.. did you enjoy it? back to work! *Whip******
One little 'trick' I've learnt over the years is that whenever you start thinking "But that's not possible" or "Wtf? Is the CPU broken"..it's time to look at your most trusted code. The bits you've been ignoring because you 'know' they are fine.
Dunno if that applies here but it does sound like it.
--
(Battle hardened industrialist)
[Brackley, UK]
My budgie can say "ploppy bottom". You have been warned. |
kuromei
|
Posted - 2007.09.16 07:45:00 -
[115]
As a doctorate candidate in computer science and math, I painfully understand how difficult this bug must have been to find. However, leaving open ended conditionals will always be prone to throwing this error. You can never know the full set of values users will pass to them, and therefore refraining from "else {" is logically guarenteed to ensure this problem doesn't cascade, which makes it incredibly easy to prevent and troubleshoot.
It's also one of the downsides with using a nonstrict programming language, but honestly the benefits usually outweight the negatives.
Anyway, I appreciate your communication, and your vigilance is solving it. As usually, you guys are the best!
|
Setana Manoro
Gallente
Firefly Inc.
|
Posted - 2007.09.16 07:55:00 -
[116]
Damn, talk about elusive. Nice job. :)
I'm not neutral in RAGOON vs BOB, I just dislike both sides. :) |
Farscape Hw
Black Omega Security
Pandemic Legion
|
Posted - 2007.09.16 08:09:00 -
[117]
i lost 500billion isk, 5 titans and a 2008 chevy silverado in the database issue. please reimburse me.
|
Mr Feltcher
|
Posted - 2007.09.16 08:24:00 -
[118]
well done ccp, good job on keeping the severs limping along and good work on fixing the crashes..
|
Leia Corrino
Essence Shipyards
Essence Enterprises
|
Posted - 2007.09.16 08:48:00 -
[119]
Originally by: CCP Explorer
Originally by: Swamp Ziro
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
Yes, seriously.
Dear God!
Thank you all CCP staff for your hard work, but don't risk you health like that!
/me sends extra Kudos to CCP John Proctor
|
Kastaldi
Caldari
Science and Trade Institute
|
Posted - 2007.09.16 08:58:00 -
[120]
Well, thanks for all this debugging job. I'm wondering where are all those players crying for their precious bucks lost in all these crashes... They seem vanished... :)
|
Longword
Cirrius Technologies
O X I D E
|
Posted - 2007.09.16 09:02:00 -
[121]
Many thanks to CCP for keeping us in the loop on this one, it means a lot to know what's going on when we can't get our daily fix of gametime. Full marks for customer communication.
|
Ab Tallen
The Dead Parrot Shoppe Inc.
Brutally Clever Empire
|
Posted - 2007.09.16 09:07:00 -
[122]
Ouch.
Good job on finding that one. Get some sleep
|
Rin Kaeda
Underground-Operators
|
Posted - 2007.09.16 09:27:00 -
[123]
Astounding job CCP!
I hope John is ok, never a good decision to take those long sessions :) (done it myself when I was younger so I know how the body reacts.)
|
Svavz
Gallente
North Face Force
|
Posted - 2007.09.16 09:38:00 -
[124]
Now while I am incredibly greatful that this has finally been fixed... You have quite clearly claimed it is your fault.
Will their be any sort of reimbursement or will the EULA "we can do wtf we want" be used ?
I don't mean to come across as harsh, just before it was fixed everybody was asking without reply.
|
JohnnyGoofy
|
Posted - 2007.09.16 10:08:00 -
[125]
Thanks for the detailed description, and good work guys.
Bugs like this happens from time to time in a constantly changing environment no matter how much you want to avoid it.
Hope you have a speedy recovery John.
/JohnnyGoofy
|
Vrabac
Amarr
BALKAN EXPRESS
|
Posted - 2007.09.16 10:17:00 -
[126]
That's what we've been waiting for all this time. Good work, altough not as fast as many would like.
|
necronarcosis
Fury Corporation
|
Posted - 2007.09.16 10:19:00 -
[127]
A extremly hard job you guys have, but ultimately very rewarding.
Many thanks for your perseverance and patience with the "vocal community". 
and a silent thanks from those of us that knew you would pull this one out of the fire.
|
Sylthi
Minmatar
Coreward Pan-Galactic
|
Posted - 2007.09.16 10:48:00 -
[128]
An ounce of prevention is often worth a pound of cure. Be more careful when modifying existing procedures, ESPECIALLY when you monkey with something that affects the whole system. But, hindsight is often clearer than foresight, so that isn't always easy; and you did admit that it was YOUR mistake and take (slow) steps to fix it.
So, all in all, I agree that modest thanks are in order for a decent job fixing the problem. So, here goes:
"Thank you for doing what you get paid to do, and obviously wish to continue getting paid to do. HOPEFULLY this will allow us to get back to normal play. Cheers."
*
* |
Shari Vegas
Minmatar
Ctrl Alt Elites
|
Posted - 2007.09.16 11:09:00 -
[129]
That's great and all, but now where's our ******* reimbursment?!
I kid. Good work on this and great communication, unlike certain other game companies who take their forums offline whenever there's a problem and never give an explanation of what happened that caused their customers whatever annoyance there was. While the problems and server downages were certainly annoying, this thorough explanation of the problem and what happened makes me feel even better about keeping up my subscriptions.
And to all the people who say they're going to buy John Proctor a drink, my god people, are you trying to put him back in the hospital when he needs a new liver?
Originally by: CCP Wrangler
I have no clue.
|
Zenst
Gallente
Aliastra
|
Posted - 2007.09.16 11:28:00 -
[130]
Thank you for the details insite into the problem - much apprecieated.
Remember the coders rule:
Rule #1 There is no excpetions
Rule #2 For every rule there is an exception
Rule #3 For every exception there is a rule.
If you obey all those rules you will have no bugs - welcome to the real World :).
|
Alessandraa
|
Posted - 2007.09.16 11:31:00 -
[131]
Yay =)
Congrats to all insomniacs involved in fixing the problems, and thanks for keeping us up to date with detailed information on the issue.
Sleep well!
|
Maximada
Minmatar
Silent Scream
|
Posted - 2007.09.16 11:44:00 -
[132]
nice one dev team.
I read the post whist heavilly intoxicated and there was an implosion somewhere inside my head, then i wondered if life was really worth living.
|
Postlatta Mouseanon
Caldari
State War Academy
|
Posted - 2007.09.16 11:47:00 -
[133]
Originally by: CCP John Proctor
I can go sleep now
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
Ya know, I'm in IT as well and have pulled 48 hour shifts in emergencies.
But 48 hours is my limit. I leave. Whether I fixed it or not. I did 80 hours straight once, and ended up sick. I probably should have gone to the hospital. So I never work more than a 48 now.
On one hand, I'd like to say thanks for the work.
On the other hand, WTF were you thinking? Did you get a nap?
I've set policy in my department that if you work more than 20 hours, you are encouraged to crash on a cot in the datacenter for at least 2 hours.
May I suggest that CCP buy some cots?
I'd also like to point out that while I've complained about the server uptime, I don't want anyone sick over it.
|
Devious
|
Posted - 2007.09.16 11:49:00 -
[134]
Oh woot they fixed the problem i diagnosed like a week ago, yeah i sent a in game petition saying there was an issue with sov in certain systems and the server couldnt assign codes as it was crashing and reverting back to old database cache's.... you kinda know that when you see stuff in system not despawning, player tags in local that are not there etc etc...that was a week ago before this stuff started happening, when they where umming and ahhing about hardware faults. CCP needs to get a Real server team not some two bit 10 man gang that end up near killing themseleves from under funded ISK leeching vampires...anyway nice work about time, lets hope the next bad bit of coding is easier to fix...oh and compensation would be nice...an event or something in game, not this rubbish alliance only tournies something everyone can do no matter how long they been in game....
|
Acoco Osiris
Gallente
Sublime.
|
Posted - 2007.09.16 11:57:00 -
[135]
/me skips most of page 4 and 5, being too impatient to read it all
I can't quite understand some... actually most of what was said. But I do feel the pain-in a computer programming course (semester-long high school, pretty boneheaded stuff, got over 100% final grade), I had a final project to do. It kept crashing the computer, and I couldn't figure out why... until I realized that in 35 pages of C++, I'd forgotten to change a 1 to a 4 . I have little doubt that you guys have at least 35,000,000 pages of code to sift through. Good job, thanks for getting Tranquility up again, and generally speaking, work tends to stop when you go to the hospital.
------------------------------
One more soldier off to war...
And one Velator in my hangars. |
Ilmonstre
Nymphetamine Junkies
|
Posted - 2007.09.16 12:01:00 -
[136]
i will be happy with a full week of no crashes of TQ
|
Cletus Graeme
Caldari
Bladerunners
Mordus Angels
|
Posted - 2007.09.16 13:02:00 -
[137]
Edited by: Cletus Graeme on 16/09/2007 13:02:29
That's the most honest, accurate, full explanation I've ever been given of a technical problem from Tech Support and given that I work in I.T myself I can really appreciate:
(1) The complexity of the issues and the difficulty in finding a solution
(2) The dedication required to track down the multiple problems and fix them
(3) The goodwill behind the detailed explanation (which admittedly not everyone will fully understand but still explains the problem as fully as possible which is something most companies would never do)
Top marks for effort & communication with this. With the latest patches finally startting to address the performance problems in EvE I'm really coming to love CCP again for making this game what it is.
Keep up the good work! But take a break first - you've earned it!
|
|
CCP Wrangler
|
Posted - 2007.09.16 13:27:00 -
[138]
When it comes to reimbursing ship losses and similar during this period of crashes that's not something we can comment on in this thread. As with all reimbursement it will be handled by our Customer Support people on a case by case basis.
As has been said, it was never the intention that conquerable stations should be able to be set at capital stations. In the cases where that has been done we ask the alliances affected to work with out Customer Support team to resolve this issue.
The 3asy company alliance appears not to have paid its bill, the crash revealed that and the alliance was disbanded. You will have to recreate the alliance to fix this, however we will help you out of you have any problems recreating the alliance and in case you had a alliance logo made we will make sure you get it back as soon as we can. That will be handled through the petition you made though.
I hope this has answered the recent questions in this thread.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Guvnor RBM
Welsh Commanders
|
Posted - 2007.09.16 13:47:00 -
[139]
Edited by: Guvnor RBM on 16/09/2007 13:49:30
Thanks for that, I checked but we didn't get the bill...
The cheque is in the post ect ect :)
Any way, as long as it can be sorted
If some one could contact me in game or 101 commander to get it running again it would great
Thanks
|
|
CCP Wrangler
|
Posted - 2007.09.16 14:01:00 -
[140]
Originally by: Guvnor RBM
Edited by: Guvnor RBM on 16/09/2007 13:49:30
Thanks for that, I checked but we didn't get the bill...
The cheque is in the post ect ect :)
Any way, as long as it can be sorted
If some one could contact me in game or 101 commander to get it running again it would great
Thanks
The bill is sent to the CEO of the Executor Corporation for the Alliance.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Svavz
Gallente
North Face Force
|
Posted - 2007.09.16 14:02:00 -
[141]
Edited by: Svavz on 16/09/2007 14:01:57
Originally by: Guvnor RBM
Edited by: Guvnor RBM on 16/09/2007 13:49:30
Thanks for that, I checked but we didn't get the bill...
The cheque is in the post ect ect :)
Any way, as long as it can be sorted
If some one could contact me in game or 101 commander to get it running again it would great
Thanks
You need to pay the 1billion again dude :P
Edit: Wrangler! My post was in regards to suscription reimbursement...
|
Terianna Eri
Amarr
STK Scientific
M. PIRE
|
Posted - 2007.09.16 14:03:00 -
[142]
Thanks CCP <3
That database stuff can get tricky sometimes, I imagine
__________________________________
|
ViolenTUK
Gallente
Vindicated Exiles
|
Posted - 2007.09.16 14:07:00 -
[143]
Originally by: Svavz
Edited by: Svavz on 16/09/2007 14:01:57
You need to pay the 1billion again dude :P
Edit: Wrangler! My post was in regards to suscription reimbursement...
He knows that the billion will need to be paid again thats why he said the cheques in the post as in he is waiting for CCP to send him the billion that he paid when he started the alliance 6 months ago as this is CCPs mistake.
www.eve-players.com |
Svavz
Gallente
North Face Force
|
Posted - 2007.09.16 14:11:00 -
[144]
Originally by: ViolenTUK
Originally by: Svavz
Edited by: Svavz on 16/09/2007 14:01:57
You need to pay the 1billion again dude :P
Edit: Wrangler! My post was in regards to suscription reimbursement...
He knows that the billion will need to be paid again thats why he said the cheques in the post as in he is waiting for CCP to send him the billion that he paid when he started the alliance 6 months ago as this is CCPs mistake.
he wont get it though :P
|
ViolenTUK
Gallente
Vindicated Exiles
|
Posted - 2007.09.16 14:14:00 -
[145]
Dont be so negative.
www.eve-players.com |
Marcus DeMon
|
Posted - 2007.09.16 14:17:00 -
[146]
The biggest issue we have is not the ISK as such... BUT THERE IS NO BILL ! for us to pay :(
|
Kropotkin
Gallente
Center for Advanced Studies
|
Posted - 2007.09.16 14:25:00 -
[147]
Originally by: CCP Explorer
Originally by: Swamp Ziro
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
Yes, seriously.
Respectfully suggest you add "Remove Blackberry" to your checklist for "Sending Colleague to Hospital".
And does this sort of thing not trigger an Investigation by Occupational Safety and Health authorities?
|
|
CCP Wrangler
|
Posted - 2007.09.16 14:42:00 -
[148]
Originally by: Marcus DeMon
The biggest issue we have is not the ISK as such... BUT THERE IS NO BILL ! for us to pay :(
The bill wasn't paid, and because of that the alliance disbanded. However, Guvnor RBM has a petition in and this will be dealt with there.
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Guvnor RBM
Welsh Commanders
|
Posted - 2007.09.16 14:44:00 -
[149]
Originally by: CCP Wrangler
Originally by: Marcus DeMon
The biggest issue we have is not the ISK as such... BUT THERE IS NO BILL ! for us to pay :(
The bill wasn't paid, and because of that the alliance disbanded. However, Guvnor RBM has a petition in and this will be dealt with there.
I await my responce, thanks
|
|
CCP Wrangler
|
Posted - 2007.09.16 14:46:00 -
[150]
"Powered by Stella Artois"??
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Guvnor RBM
Welsh Commanders
|
Posted - 2007.09.16 14:52:00 -
[151]
I can only PvP when drunk... very very drunk
|
ViolenTUK
Gallente
Vindicated Exiles
|
Posted - 2007.09.16 14:56:00 -
[152]
Originally by: Guvnor RBM
I can only PvP when drunk... very very drunk
Very common in the 3asy Company Alliance.
www.eve-players.com |
Guvnor RBM
Welsh Commanders
|
Posted - 2007.09.16 15:00:00 -
[153]
Fix 3asy Company and the Dev's can join us on a "mission"
You Tubeage
|
Zuko Droner
|
Posted - 2007.09.16 16:25:00 -
[154]
I don't know where this topic ended up, but good job finding that bug.
|
Jordan Musgrat
Convergent
Firmus Ixion
|
Posted - 2007.09.16 17:19:00 -
[155]
Thanks for fixing this. For our sanity and your hospital bill's sake, I hope this never happens again. (If you don't have to pay hospital bills I dun wanna hear about it)
-----------
Primary is family values, secondary is 0.0... |
Lester Dean
Gallente
Vanguard Frontiers
INVICTUS.
|
Posted - 2007.09.16 17:58:00 -
[156]
Bet the cto had a hell of a week. Great job on the fix CCP :
|
iqplayer
Caldari
Dragon's Rage
Intrepid Crossing
|
Posted - 2007.09.16 18:01:00 -
[157]
Edited by: iqplayer on 16/09/2007 18:01:25
Originally by: Guvnor RBM
I can only PvP when drunk... very very drunk
Is that why your name is spelled differently in your signature (Guvnor RMB) than in your character name below your portait? (Guvnor RBM)
|
Guvnor RBM
Welsh Commanders
|
Posted - 2007.09.16 18:10:00 -
[158]
Originally by: iqplayer
Edited by: iqplayer on 16/09/2007 18:01:25
Originally by: Guvnor RBM
I can only PvP when drunk... very very drunk
Is that why your name is spelled differently in your signature (Guvnor RMB) than in your character name below your portait? (Guvnor RBM)
The sig maker has disaperred, but yeah, I didnt notice my self for almost 6 weeks
|
|
CCP Wrangler
|
Posted - 2007.09.16 18:14:00 -
[159]
Originally by: Guvnor RBM
The sig maker has disaperred, but yeah, I didnt notice my self for almost 6 weeks
Probably due to Stella...
Wrangler
Community Manager
EVE Online
Contact Support - Contact Moderators - Report Bug - Submit News Leads - Knowledge Base
Player Guide - Policies - Join ISD - Fan Submissions - DevFinder LiteÖ |
|
Vossejongk
Blake Industries
|
Posted - 2007.09.16 18:33:00 -
[160]
Originally by: torswin
http://www.yonkis.com/mediaflash/1000000.swf
thankyou! (same as wrangler (i think) posted but without the fleshlight ad).
Yonkis "Thanks you" flash is soooooooooooo hilarious lol...
CCP thanks for fixing :D
|
SheriffFruitfly
Caldari
Science and Trade Institute
|
Posted - 2007.09.16 18:53:00 -
[161]
Shorter ccp: Programmer error. Heh. If that really is all it was, good to know.
|
Umbriele
Gallente
Natural Inventions
|
Posted - 2007.09.16 20:15:00 -
[162]
In other words, an new content of the game caused more bugs.
|
Capitan Kidd
|
Posted - 2007.09.16 21:56:00 -
[163]
Is there any way to be reimbursed for training time? My other account is having issues that I think go along with the database issue. I cant renew my subscription and I am loosing valuable training time
|
Archesh
|
Posted - 2007.09.17 02:52:00 -
[164]
Why do bugs always come down to a dangling pointers or unitialized variables. I think it must be one of gods cruel jokes!
|
Hakashi
|
Posted - 2007.09.17 03:45:00 -
[165]
Hello
Few days ago, i downloaded all the SQL tables. After see what you did with those tables i guess one thing... You have a great problem !!!
Did you call to microsoft and they help you with your connections to not LOCK records in procedures ?
They told you some about the NO LOCK statement ?
hehehehe... i like this game, but im still laughing about DataBases.
|
ADSikst
|
Posted - 2007.09.17 07:05:00 -
[166]
I am appeased. :) I just wanted details, some sign of action. GREAT JOB! Keeping us informed is the way (managing client expectations and such).
|
warpod
Amarr
One Man Corp
|
Posted - 2007.09.17 07:17:00 -
[167]
Whoa.. I want to work @ CCP
|
Bohoba
Caldari
Dragons United
Pure.
|
Posted - 2007.09.17 08:46:00 -
[168]
Good Catch CCP and wow 10 stations did all that lol ok MSsql is a bugger on that sort of thing for sure.
The Dark Force is strong in EvE But it will fail
Get Into the Game it makes it fun for all |
Gunner
|
Posted - 2007.09.17 09:40:00 -
[169]
Well ofc I'm pleased the server is not crashing anymore, but I also call this very poor version management. I mean, code gets changed on Monday, database crashes from that time on and you guys don't even have a clue where to look. Personally I would have started at the latest changes and worked my way back, but that's just me, I don't run a large company with 200k subscribers to their software. Get someone to keep track of what you're doing please.
Cheers, Gunner.
|
Reem Fairchild
|
Posted - 2007.09.17 10:08:00 -
[170]
Originally by: ADSikst
I am appeased. :) I just wanted details, some sign of action. GREAT JOB! Keeping us informed is the way (managing client expectations and such).
To be honest, I think it was utterly stupid of anyone to think in that situation that there wasn't sufficient action taken by CCP, whether given details or not.
|
Seleia O'Sinnor
|
Posted - 2007.09.17 10:55:00 -
[171]
We are all addicts. No wonder we all get angry when our toys are taken from us for an undefined amount of time.
@CCP you are doing a fine job! There are lots of companies that don't provide such detailed information on their problems. That is highly satisfying. You guys won't lose me as a player .
Never play on patch, just watch.
|
Imhotep Khem
Madhatters Inc.
M. PIRE
|
Posted - 2007.09.17 12:10:00 -
[172]
I always love it when I find a big this complex. Especially when it is discovered that it is in a few extra places and you got to fix some things that you really did not notice were broken. But also especially when you did notice something was broken but it was low on the priority list.
On a different note... Are you folks using OO programming techniques? I can't believe a session could have ever been left open 
_________
If your not dyin' your not tryin'. |
FireFoxx80
Caldari
E X O D U S
Imperial Republic Of the North
|
Posted - 2007.09.17 19:50:00 -
[173]
As someone who does some dabbling with SQL Server, all I can say is ouch! Not closing transactions is a good way of killing your server. Fortunately most of my work is done in a single block of SQL. I think we only use transactions in 4-5 places (of ~50 applications).
What I do the rest of the time - Vote for a Jita bypass! |
Jenni Concarnadine
|
Posted - 2007.09.18 07:23:00 -
[174]
Thank you, both for solving the problem and also for explaining how it came to be.
It at least lets us know that the hamsters didn't bite the cables.
* * * * *
Hamsters don't overload very well |
N'jo Kampala
|
Posted - 2007.09.18 13:01:00 -
[175]
Originally by: Sleepkevert
Originally by: CCP Explorer
Originally by: Swamp Ziro
Originally by: CCP Explorer
Originally by: CCP John Proctor
Dont go 72 hours with no sleep... you get sent to the hospital and have to answer emails with your black berry from the emergency room
I wasn't sure which worried me more, that you were sent to the hospital or that you kept answering your e-mails from the ER. Thank you for keeping TQ up this week.
Wait.. seriously?
Yes, seriously.
This shows how far some CCP employees will go for their game.
♥ you CCP
But dont ever do that again...... We want you alive not dead ;-) remember that ;-)
|
Chiggie
|
Posted - 2007.09.18 13:38:00 -
[176]
Edited by: Chiggie on 18/09/2007 13:39:08
So, we, the customers were right. CCP's poor programming initiated the server crashes. Even worse it was a on-the-fly fix that didn't work. Cowboys destroy servers. I see this happen all the time in my professional life. You cannot mess with a DB of this size and simply expect it to work. Get rid of MySQL and switch to real, functional, DB such as Oracle. Hopefully get rid of this terrible lag that we get. I have yet to play this game with more than 30k people online and have it work. They lag is simply unacceptable and killing this game.
Also the returned petitions I have seen for ship reimbursements refused because YOUR damned DB crashed is out of line. CCP needs to step up to the plate and take care of of it's customers in times like this. Especially since it's all your fault. Your people who take care of petitions act as if THEY are paying for the reimbursements out of their own character wallets. I see this time and time again. And frankly, I'm getting very tired of this crap.
A very ****ed off ignored CCP Customer.
Chiggie
|
Alois Hammer
Minmatar
Hammers Slammers
Slammer's Republic
|
Posted - 2007.09.19 14:46:00 -
[177]
this was a "lemmony snickets " malfunction ......
|
Toramt
|
Posted - 2007.09.19 22:21:00 -
[178]
It is MS SQLServer, not MySQL.
|
Adel Sorra
Gallente
Steel Beasts
Cold Steel Alliance
|
Posted - 2007.09.20 22:14:00 -
[179]
Originally by: flakeonium
horror week for devs i guess yes, but could someone tell CCP John Proctor he does not have to risk his health for me ?
(i assume he is alright now)
Not you.. Eve
|
Adel Sorra
Gallente
Steel Beasts
Cold Steel Alliance
|
Posted - 2007.09.20 22:17:00 -
[180]
Originally by: Toramt
It is MS SQLServer, not MySQL.
Imagine Eve running on MySql XD
|
Sky Grunthor
Minmatar
MKS Directorate
|
Posted - 2007.09.21 19:52:00 -
[181]
Originally by: kuromei
Edited by: kuromei on 16/09/2007 07:56:37
As a doctorate candidate in computer science and math, I painfully understand how difficult this bug must have been to find. However, leaving open ended conditionals will always be prone to throwing this error. You can never know the full set of values users will pass to them, and therefore refraining from "else {" is logically guarenteed to ensure this problem doesn't cascade, which makes it incredibly easy to prevent and troubleshoot.
It's also one of the downsides with using a nonstrict programming language, which from the error report, you guys obviously are, but honestly the benefits usually outweight the negatives.
Anyway, I appreciate your communication, and your vigilance is solving it. As usual, you guys are the best!
to clarify this.. it has nothing to do with the "Programing Language" they are using in this particular instance (i.e. Stackless Python)... It was database related. Database interactions are their own little (well large) specialty and have their own plethora of problems that occur.
MKS Directorate Home
|
James Duar
Merch Industrial
GoonSwarm
|
Posted - 2007.09.22 12:42:00 -
[182]
Originally by: Sky Grunthor
Originally by: kuromei
Edited by: kuromei on 16/09/2007 07:56:37
As a doctorate candidate in computer science and math, I painfully understand how difficult this bug must have been to find. However, leaving open ended conditionals will always be prone to throwing this error. You can never know the full set of values users will pass to them, and therefore refraining from "else {" is logically guarenteed to ensure this problem doesn't cascade, which makes it incredibly easy to prevent and troubleshoot.
It's also one of the downsides with using a nonstrict programming language, which from the error report, you guys obviously are, but honestly the benefits usually outweight the negatives.
Anyway, I appreciate your communication, and your vigilance is solving it. As usual, you guys are the best!
to clarify this.. it has nothing to do with the "Programing Language" they are using in this particular instance (i.e. Stackless Python)... It was database related. Database interactions are their own little (well large) specialty and have their own plethora of problems that occur.
No it was a database issue being caused by the way a routine in the programming language was being handled, which I think was his point.
He wasn't ragging on Python, so much as commenting on the tendency of people to use generic "everything else" handlers, which are fine mostly but best practice would be to know exactly every value your routine will get.
|
Amarus
Minmatar
Dinky Core Miners
|
Posted - 2007.09.26 23:18:00 -
[183]
Has there been any news on what solution CCP plan to offer alliances affected by this problem? Total silence isn't helping.
|
| |
|
| Pages: 1 2 3 4 5 6 7 :: [one page] |