Legacy code?
I'll make a prediction. The channel you used for campaign logging was used in the past for doing something else. You thought that code was removed, but some part of it still remains. When you started campaign logging, some old code woke up, tried to do something related to that channel, and "bad things" resulted.

Edit: CCP, you do not really answer one question we all kept asking over and over.

Why not roll back while you worked the issue? In the blog you stated you stated "Our test of the rollback was confirmed to work, but we still didnGÇÖt believe the code to be the issue". But, so what? Why did you let this belief stop you from doing a rollback and letting us get on the server?

I just do not see the link here. I see your thought: "we still didnGÇÖt believe the code to be the issue", and the result: No roll back, but I do not understand your reasoning for letting that thought get that result. What was your reasoning?

We usually get pizza from Castellos yeah. It doesn't actually deliver to our area unless it's for us :p

CCP Masterplan, I get that. Thanks for the reply. But why not keep going and do the startup? Did you have some "one way" DB changes with this update that would have taken extra effort to revert or restore? So much effort that, at any given time, it looked better to just keep trying to fix the issue rather than get de-railed trying to roll back?

Also, it looked like you found the temporary fix by experimenting on TQ, something you would not have been able to do if you had done the roll back.

I expect it was also a case of "We can't reproduce this reliably on our test servers, so we have to debug it in production". I think it was somewhat unclear whether the DB was modified in a way which would've prevented the rollback, and as Masterplan said, a DB restore takes time - I seem to recall a figure of multiple hours.


CCP: Thanks for fixing it, for the skillpoints, and also for the well-written report on the pizza. <3

Best Dev Blog ever :)

I expect it was also a case of "We can't reproduce this reliably on our test servers, so we have to debug it in production". I think it was somewhat unclear whether the DB was modified in a way which would've prevented the rollback, and as Masterplan said, a DB restore takes time - I seem to recall a figure of multiple hours.


CCP: Thanks for fixing it, for the skillpoints, and also for the well-written report on the pizza. <3

Illuminating. But worryingly, I got the distinct impression that CCP figured out what was causing the problem but not why - unless the underlying cause of the logging issues has since been determined?