| Author |
Thread Statistics | Show CCP posts - 15 post(s) |
|
CCP Sharkbait
   |
Posted - 2007.03.22 12:16:00 -
[1]
we had problems during startup. we changed the node configuration to try and address the market problem that we are currently experiencing. we have had to reboot the database server now, startup estimate is 12:30
will keep you posted
oh and fixing the market is our main focus atm
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 12:29:00 -
[2]
starting up now. 8 mins or so left.
if it fails this time i will have to extend the downtime til 13:00
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 12:41:00 -
[3]
we are screwed. delaying startup again. soon as i have time i will fill you in on the details
Startup EST : 13:30
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 13:00:00 -
[4]
things are looking good.
target is still 13:30
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 13:01:00 -
[5]
 Originally by: Johncrab
Pacth full of bugs, databse failures, nodes down, reboots, extended downtimes... disapointing.
we are addressing all these issues. they are on the top of the "get done now or your fired" list
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 13:17:00 -
[6]
server is starting now. 8mins or so left. it will be in VIP mode for abit til i check it's all ok
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 13:24:00 -
[7]
having to reboot again.
EST : 13:40
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 13:45:00 -
[8]
so sry bout this. we are looking into it.
EST : 14:00
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 14:07:00 -
[9]
think the problem is found. last startup now.
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 14:21:00 -
[10]
there is a problem we believe with 1 of the machines that is stoppng the cluster from starting. we have disabled it now and trying to start up
EST : 14:30
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 14:29:00 -
[11]
all the machines are called solnodes. it was machine 26 that appears to be broken. startup in 1 min, all is looking good.
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 15:04:00 -
[12]
i'm busy atm, but i promise i will give you a report about it asap.
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 16:40:00 -
[13]
ok, it wasn't just me involved in making it work. it 3-4 of us all working together. i'm just the face you see because i'm not as ulgy as the others.
basically we put out a fix for the market problems, which worked perfectly on sisi but for some reason stopped the market even starting on TQ. this issue is being looked into right now. after trying to start the server a few times, we decided to role the fix back and run with the older code.
while trying to start up the server with the old code, 1 of the servers kept dying, but having over 100 servers, it's hard to find out which 1. once we found out the problem server, it was removed from the cluster and the cluster started up 1st time.
now there is still an issue with the market leaking memory serverside. this is currently our top focus, but as a temporary fix we have changed the market to use 30 nodes instead of only 4. this basically means we have 72gig of ram instead of 8gig. so there is now plently of memory for the market to eat, but this does not mean the problem is being ignored.
i would like to say sry bout all of this and thx for the support, it does help us tbh. even if it is only words 
thx peeps
|
|
|
CCP Sharkbait
|
Posted - 2007.03.22 19:58:00 -
[14]
just to follow this up. we have a possible fix for the market issues on TQ running on sisi over night. if you can please log in and test the basic market functions.
if i see nothing wrong with the market node in the morning and there are no errors in the logs, then the fix will be on TQ at downtime tomorrow.
|
|
|
CCP Sharkbait
|
Posted - 2007.03.23 08:11:00 -
[15]
Originally by: Nalar Marnith
I appreciate the information that's come out with this problem. Normally CCP keeps us in the dark, and although I missed a night's gaming (being an aussie and not having that many nights to play), I don't mind as much knowing WHY.
Being a programmer of complex apps myself, I know that things can work GREAT on the test machine and fail in a big fat heap on production. This is using identical hardware, I can't even imagine the issues deploying to a cluster.
i have not been here long, i was a player for 3 years before i started at CCP and i really hated it when there was no information, only a comment "server down". nothing is more anoying.
i decided to tell all information as i get it and as yet there is some people that think i shouldn't say everything i do, but you peeps (players) like it, Oveur (Senior Producer) likes it and Hellmar (CCP CEO)likes me doing it. so if those 2 don't stop me doing it, i will continue on opening my big fat mouth
|
|
| |
|