Quote:

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

Originally by: Dark Shikari

---

Edited by: Dark Shikari on 19/10/2007 22:51:56
First \o/
Thanks for the information... its good to see that CCP is communicating, even as people spread dozens of false rumors claiming doom.

---

Originally by: Dark Shikari

---

Edited by: Dark Shikari on 19/10/2007 22:53:01
First \o/

Thanks for the information... its good to see that CCP is communicating, even as people spread dozens of false rumors claiming doom.

Some rumors I've seen, for laughs:

- Cluster hacked, rollback for obvious reasons.
- Terrorist attack (ranged from bombs on subways to nuclear explosion) on London, TQ servers wrecked
- All sovereignty data reset meaning every single system is back to Sov1 on day 1 including NPC space.
- All wallet data lost, CCP will compensate this by giving everyone one free day of EVE.
- BoB's towers in FAT-GP were coming out of reinforced, CCP crashed the servers to protect them.
- Also, the rollback is for...
-- EVE-TV
-- EVE-O, Dark Shikari is devastated

---

Originally by: CCP Wrangler

---

Booooo!!! Tarminic sux!!!

---

Originally by: Kyoto Rose

---

Edited by: Kyoto Rose on 19/10/2007 22:53:45
Note: I was not around to personally confirm this information. Just spreading rumors potential rumors. It was posted many hours ago, before the servers were online or an official response was posted. Pretty close to what little information we received on the issue:

Quote:

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

---

Originally by: Dark Shikari

---

First \o/

Some rumors I've seen, for laughs:

- BoB's towers in FAT-GP were coming out of reinforced, CCP crashed the servers to protect them.

---

Originally by: Caine 607

---

Edited by: Caine 607 on 19/10/2007 22:54:52
Cailais.. I love that sig... best I've ever seen !

p.s. yours too Neth !

---

Originally by: CCP Wrangler

---

During this downtime a lot of rumors were spread, this one among them. I can assure you that our Sharkbait did not post this, but he is very eager to find out who did...

---

Originally by: Liz Kali

---

Tic Toc Tic Toc , time is ticking

---

I owned someone on forums!!!

Originally by: Graisse

---

Who benefited from the hack?

---

Quote:

---

6539 officer items on one account

---

Originally by: CCP Wrangler

---

Originally by: Kyoto Rose

---

Edited by: Kyoto Rose on 19/10/2007 22:53:45
Note: I was not around to personally confirm this information. Just spreading rumors potential rumors. It was posted many hours ago, before the servers were online or an official response was posted. Pretty close to what little information we received on the issue:

Quote:

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

---

During this downtime a lot of rumors were spread, this one among them. I can assure you that our Sharkbait did not post this, but he is very eager to find out who did...

---

Originally by: Cailais

---

Originally by: Dark Shikari

---

Edited by: Dark Shikari on 19/10/2007 22:51:56
First \o/
Thanks for the information... its good to see that CCP is communicating, even as people spread dozens of false rumors claiming doom.

---

How does he do that? Uncanny.

C.

---

Originally by: CCP Wrangler

---

Originally by: Kyoto Rose

---

Edited by: Kyoto Rose on 19/10/2007 22:53:45
Note: I was not around to personally confirm this information. Just spreading rumors potential rumors. It was posted many hours ago, before the servers were online or an official response was posted. Pretty close to what little information we received on the issue:

Quote:

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

---

During this downtime a lot of rumors were spread, this one among them. I can assure you that our Sharkbait did not post this, but he is very eager to find out who did...

---

Originally by: Empire marketslave

---

Originally by: Graisse

---

Who benefited from the hack?

---

they people at CCP who had to work over time to fix it, several hours of OT


or thier spouces that hate thier guts and was glad that they were out of the house

---

Originally by: shinsushi

---

I have this crazy idea to prevent the massive amount of whining when EvE has unscheduled downtimes. Skill Queues.

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Originally by: shinsushi

---

I have this crazy idea to prevent the massive amount of whining when EvE has unscheduled downtimes. Skill Queues.

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Originally by: Skywalker

---

CAN WE GOT ANOTHER ARMAGEDDON DAY AS COMPENSATION ?
(At another time of day)

---

Originally by: Mel Ionix

---

Two very obvious things CCP should learn from this:

1) Noone had a ****in clue what was happening, rumours were rife. You NEED a way to communicate with customers in situations like this, i.e. put the forums on a seperate server

---

Originally by: CCP Spielmann

---

What we discovered was an indication that one of our databases was being accessed through a security breach. While some may feel that such a drastic reaction was not warranted, it is always our approach to err on the side of caution in order to protect the players.



We can also confirm that no personal details such as usersÆ credentials or credit card numbers were exposed through this incident.

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Quote:

---

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall

---

Originally by: elider

---

-Then he magically knows the internal structure of your networks and finds the database server

---

Originally by: elider

---

-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

---

Originally by: elider

---

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Originally by: El'essar Viocragh

---

Originally by: shinsushi

---

I have this crazy idea to prevent the massive amount of whining when EvE has unscheduled downtimes. Skill Queues.

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Train longer skills :D

Knowing your skill won't finish another 24 days makes you feel comfy.

PS: and just to reiterrate from the other thread. Nice work guys, ONLY viable option in such a case. And I'm glad you had the balls to do it. Wouldn't want it any other way.

---

Originally by: Akita T

---

Still, over nine hours of downtime ? Bloody hell...

SKILL QUEUE

It's on the "to do" list since, like, forever.
Just do it already, ffs. It's not like it's rocket science...

---

Originally by: Schalana

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Of course, this couldn't of happened could it....

Relational databases don't make sense to hackers, firewalls are totally in passable. If you think this is true, you've not got enough knowledge to comment.

CCP did EXACTLY what they should of, imagine leaving this until the next d/t, the damage that could of been done. Also, in repsonse to the 'weekly' backups comment, I'm hoping that they are on some sort of continuous backup system?

---

Originally by: dev1lwoman

---

REPOSTED FROM ANOTHER THREAD
While todays server issues are going to be discussed in another thread, I believe there is one issue that while connected deserves a thread of its own.

While there was a lot of frustration on other forums and on the open TS hosted by BE at being unable to play, that frustration at times was almost anger at the inability of CCP to communicate at all.

Surely it is about time for CCP to look at hosting the forums seperatly from the game, a lot of the anger/frustration felt and expressed today could have been avoided if CCP had the means to communicate with its customer base.

Technical problems happen in any enviroment, that is understandable, not communicating with the paying customer is not.

Please CCP seperate the forum host from the game servers so at least in the future you have a working channel to keep us informed


ADDED: The issue here ISD is not wether the details of this security breach could have been discussed, but if the forums were on a sperate server/host there could have been at least some element of communication. It is simplicity itself to create a datbase of users for the forums seperate from the main database that automatically updates as people sign up to Eve-Online. and when the main game database is down the forums would still be open for CCP to let us know as much as is possible/wise.

I have ZERO problem with the actions taken today top secure the game database, I do have issues with the total lack of communication.

A simple post saying something like...
"There are issues with the database that are being worked upon, this will take a few minutes/couple hours/we have no idea of how long but be assured we have every available person working on it etc" would have made a hell of a difference in easing the frustration some people felt.

---

Originally by: El'essar Viocragh

---

Originally by: shinsushi

---

I have this crazy idea to prevent the massive amount of whining when EvE has unscheduled downtimes. Skill Queues.

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Train longer skills :D

Knowing your skill won't finish another 24 days makes you feel comfy.

PS: and just to reiterrate from the other thread. Nice work guys, ONLY viable option in such a case. And I'm glad you had the balls to do it. Wouldn't want it any other way.

---

Originally by: Dark Shikari

---

Edited by: Dark Shikari on 19/10/2007 22:56:41
First \o/

Thanks for the information... its good to see that CCP is communicating, even as people spread dozens of false rumors claiming doom.

Some rumors I've seen, for laughs:

- Cluster hacked, rollback for obvious reasons.
- Terrorist attack (ranged from bombs on subways to nuclear explosion) on London, TQ servers wrecked
- All sovereignty data reset meaning every single system is back to Sov1 on day 1 including NPC space.
- All wallet data lost, CCP will compensate this by giving everyone one free day of EVE.
- BoB's towers in FAT-6P were coming out of reinforced, CCP crashed the servers to protect them.
- Also, the rollback is for...
-- EVE-TV
-- EVE-O, Dark Shikari is devastated

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

---

Originally by: Kyoto Rose

---

Edited by: Kyoto Rose on 19/10/2007 22:58:51

Wrangler, see here: http://www.battleclinic.com/forum/index.php/topic,12960.msg57856.html#msg57856

That's where I first saw it.

---

Originally by: elider

---

Edited by: elider on 19/10/2007 23:16:24

Originally by: Schalana

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Of course, this couldn't of happened could it....

Relational databases don't make sense to hackers, firewalls are totally in passable. If you think this is true, you've not got enough knowledge to comment.

CCP did EXACTLY what they should of, imagine leaving this until the next d/t, the damage that could of been done. Also, in repsonse to the 'weekly' backups comment, I'm hoping that they are on some sort of continuous backup system?

---

Big fan of Mission Impossible ?

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Originally by: CCP Wrangler

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

Originally by: Mel Ionix

---

Two very obvious things CCP should learn from this:

1) Noone had a ****in clue what was happening, rumours were rife. You NEED a way to communicate with customers in situations like this, i.e. put the forums on a seperate server

---

Originally by: CCP Spielmann

---

At 10:25 GMT today we discovered an anomaly in the EVE Online Database indicating a potential exploit. Our policy in such cases is to mobilize a taskforce of internal and external experts to evaluate the situation. At 12:57 that group concluded that our best course of action was to go completely dark while an exhaustive scan of our entire infrastructure was executed.

What we discovered was an indication that one of our databases was being accessed through a security breach. While some may feel that such a drastic reaction was not warranted, it is always our approach to err on the side of caution in order to protect the players.

We of course understand the effect and disruption this has had for our players and apologize for not having been able to explain to the community what was going on. In these cases it can often be counterproductive to containment to give out information while we are in the process of evaluating the scope of the problem.

Our taskforce quickly found the security breach and prevented that from being used. We subsequently found three cases of database actions being performed through the security breach but none of those affected other users than the one doing the actions. We can also confirm that no personal details such as usersÆ credentials or credit card numbers were exposed through this incident.

The servers were brought back online at 22:00 GMT and we will of course continue to monitor the situation closely over the weekend and the following week.

Again we sincerely apologize for this disruption.

Regards,
-J=n H÷r=dal
Chief Operating Officer, CCP

---

Originally by: CCP Spielmann

---

At 10:25 GMT today we discovered an anomaly in the EVE Online Database indicating a potential exploit.

---

Originally by: CCP Spielmann

---

What we discovered was an indication that one of our databases was being accessed through a security breach. While some may feel that such a drastic reaction was not warranted, it is always our approach to err on the side of caution in order to protect the players..

---

Originally by: Kayna Eelai

---

Originally by: CCP Spielmann

---

At 10:25 GMT today we discovered an anomaly in the EVE Online Database indicating a potential exploit. Our policy in such cases is to mobilize a taskforce of internal and external experts to evaluate the situation. At 12:57 that group concluded that our best course of action was to go completely dark while an exhaustive scan of our entire infrastructure was executed.

What we discovered was an indication that one of our databases was being accessed through a security breach. While some may feel that such a drastic reaction was not warranted, it is always our approach to err on the side of caution in order to protect the players.

We of course understand the effect and disruption this has had for our players and apologize for not having been able to explain to the community what was going on. In these cases it can often be counterproductive to containment to give out information while we are in the process of evaluating the scope of the problem.

Our taskforce quickly found the security breach and prevented that from being used. We subsequently found three cases of database actions being performed through the security breach but none of those affected other users than the one doing the actions. We can also confirm that no personal details such as usersÆ credentials or credit card numbers were exposed through this incident.

The servers were brought back online at 22:00 GMT and we will of course continue to monitor the situation closely over the weekend and the following week.

Again we sincerely apologize for this disruption.

Regards,
-J=n H÷r=dal
Chief Operating Officer, CCP

---

and it was not like possible, to make a www.eve-online.com/index.html (ergo: no database involved) to let us know what the hell is going on?

i am a paying customer and i can't believe you had us sitting in the complete dark FOR HOURS with zero information. we had t oaccess 3rd party places to get (unconfirmed) bits and pieces of information. things only started to light up a bit when everyone went to eve radio to get some more info.

very bad towards your customers. worse than blizzard, and that means a lot.

---

Originally by: CCP Wrangler

---

I have no clue.

---

Originally by: Red Desire

---

Originally by: CCP Spielmann

---

At 10:25 GMT today we discovered an anomaly in the EVE Online Database indicating a potential exploit.

---

This is the only thing that matters, the rest is PR.
I find hard to belive that a person "hacked" through all CCP security, not impossible.
I think that a person might found some account password which had write rights in the data base.
After he studied the data base he made some modifications,then my bet is CCP found something by mistake and they freaked out.
The question what I'm courios aboutis how did he get the password and access to the network.

---

Originally by: Father Weebles

---

Yeah ofc this happens right when my skill finished. 7 hours of training down the drain. Add a damn skill queue..for the love of all things holy

---

Originally by: Dark Shikari

---

- BoB's towers in FAT-6P were coming out of reinforced, CCP crashed the servers to protect them.

---

Originally by: CCP Wrangler

---

Originally by: dev1lwoman

---

REPOSTED FROM ANOTHER THREAD
While todays server issues are going to be discussed in another thread, I believe there is one issue that while connected deserves a thread of its own.

While there was a lot of frustration on other forums and on the open TS hosted by BE at being unable to play, that frustration at times was almost anger at the inability of CCP to communicate at all.

Surely it is about time for CCP to look at hosting the forums seperatly from the game, a lot of the anger/frustration felt and expressed today could have been avoided if CCP had the means to communicate with its customer base.

Technical problems happen in any enviroment, that is understandable, not communicating with the paying customer is not.

Please CCP seperate the forum host from the game servers so at least in the future you have a working channel to keep us informed


ADDED: The issue here ISD is not wether the details of this security breach could have been discussed, but if the forums were on a sperate server/host there could have been at least some element of communication. It is simplicity itself to create a datbase of users for the forums seperate from the main database that automatically updates as people sign up to Eve-Online. and when the main game database is down the forums would still be open for CCP to let us know as much as is possible/wise.

I have ZERO problem with the actions taken today top secure the game database, I do have issues with the total lack of communication.

A simple post saying something like...
"There are issues with the database that are being worked upon, this will take a few minutes/couple hours/we have no idea of how long but be assured we have every available person working on it etc" would have made a hell of a difference in easing the frustration some people felt.

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Originally by: shinsushi

---

Thanks for the info! I guess my 18 hour skill wasn't long enough.....

---

Originally by: Kayna Eelai

---

there is a difference between updates every 15 minutes and no info at all in over 6 hours.

---

Originally by: Madelchai

---

Something happened. That's all you need to know.

---

Originally by: El'essar Viocragh

---

Originally by: shinsushi

---

Thanks for the info! I guess my 18 hour skill wasn't long enough.....

---

In this case, it wasn't. But there will be other times when long skills end during times of server instability or unexpected downtimes. Or when a short skill is just long enough to finish right after TQ is up again.

It simply happens. Not often, but it does. And overall, does it really matter?

---

Originally by: CCP Wrangler

---

Originally by: Kayna Eelai

---

there is a difference between updates every 15 minutes and no info at all in over 6 hours.

---

Unfortunately with everything down I had to sit in IRC and ask people to post on any fansites and corp or alliance sites they could. But as I posted above we have learned from this and we'll put in some sort of system that will let us post official news somewhere in case anything like this happens again. Of course, we hope we won't ever have to use that kind of system.

---

Originally by: Kayna Eelai

---

welcome to 2005
you should really start to look what other MMO companies are doing sometimes. you have several flaws here and there and if you would just take a look at others from time to time, you would have learned this lesson LONG AGO.

sometimes i think CCP are sitting in some sort of bunker, cut off from the rest of the world, just sitting in their little universe not careing what happens outside there.

---

Originally by: High Sierra

---

note to self:
dont ever say anything to anyone on the internet about anything ever again.

---

Originally by: CCP Wrangler

---

Originally by: Kayna Eelai

---

there is a difference between updates every 15 minutes and no info at all in over 6 hours.

---

Unfortunately with everything down I had to sit in IRC and ask people to post on any fansites and corp or alliance sites they could. But as I posted above we have learned from this and we'll put in some sort of system that will let us post official news somewhere in case anything like this happens again. Of course, we hope we won't ever have to use that kind of system.

---

Originally by: Kane Rizzel

---

CCP were desperately trying to get the issue sorted, I'm sure they're very sorry they weren't able to send you a personal update on what was happening as they were a touch busy.

---

Originally by: Kayna Eelai

---

Quote:

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

welcome to 2005
you should really start to look what other MMO companies are doing sometimes. you have several flaws here and there and if you would just take a look at others from time to time, you would have learned this lesson LONG AGO.

sometimes i think CCP are sitting in some sort of bunker, cut off from the rest of the world, just sitting in their little universe not careing what happens outside there.

---

Originally by: Kayna Eelai

---

Originally by: Kane Rizzel

---

CCP were desperately trying to get the issue sorted, I'm sure they're very sorry they weren't able to send you a personal update on what was happening as they were a touch busy.

---

yeah, desperately... coz writing the same they wrote in this post, but into a INDEX.HTML and putting it online where people could see it, is soooo hard.

or a blank page just showing a link to the IRC or whatever.

when i pay MONEY for a online game, i expect more than just the pixels of the game, i expect customer service, otherwise i would just go back and play tetris, for free.

---

Originally by: High Sierra

---

note to self:
dont ever say anything to anyone on the internet about anything ever again.

---

Originally by: Kane Rizzel

---

We all pay to play, you're not a special and unique snowflake. It's a game of internet spaceships, seriously dude, chill out

---

Originally by: Kayna Eelai

---

when i pay MONEY for a online game, i expect more than just the pixels of the game, i expect customer service, otherwise i would just go back and play tetris, for free.

---

Originally by: High Sierra

---

note to self:
dont ever say anything to anyone on the internet about anything ever again.

---

Originally by: El'essar Viocragh

---

Originally by: shinsushi

---

I have this crazy idea to prevent the massive amount of whining when EvE has unscheduled downtimes. Skill Queues.

I know it sound strange, bare with me though. Allow a character to toggle a secondary skill that will automatically begin training when the primary is done. This way no one could ever complain about this stuff.

---

Train longer skills :D

Knowing your skill won't finish another 24 days makes you feel comfy.

---

Originally by: Kayna Eelai

---

We all pay to play, you're not a special and unique snowflake. It's a game of internet spaceships, seriously dude, chill out

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

Originally by: Ratzap

---

While not impossible that someone outside broke in, most breaches tend to be inside jobs or social engineering attacks on inside people. Someone coming in entirely cold is not the most likely scenario but can always happen (given enough time and determination).
It'll be interesting to see if there's any follow up posts explaining more clearly what/how (doubt it but meh, curious).

Ratzap

---

Originally by: Kayna Eelai

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

1) if you're such a security wonk, how comes your system was not secure enough?

---

Originally by: CCP Wrangler

---

Originally by: Kayna Eelai

---

there is a difference between updates every 15 minutes and no info at all in over 6 hours.

---

Unfortunately with everything down I had to sit in IRC and ask people to post on any fansites and corp or alliance sites they could. But as I posted above we have learned from this and we'll put in some sort of system that will let us post official news somewhere in case anything like this happens again. Of course, we hope we won't ever have to use that kind of system.

---

Originally by: Kayna Eelai

---

Quote:

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

welcome to 2005
you should really start to look what other MMO companies are doing sometimes. you have several flaws here and there and if you would just take a look at others from time to time, you would have learned this lesson LONG AGO.

sometimes i think CCP are sitting in some sort of bunker, cut off from the rest of the world, just sitting in their little universe not careing what happens outside there.

---

Originally by: Kayna Eelai

---

yeah, desperately... coz writing the same they wrote in this post, but into a INDEX.HTML and putting it online where people could see it, is soooo hard.

---

Originally by: Kayna Eelai

---

when i pay MONEY for a online game, i expect more than just the pixels of the game, i expect customer service, otherwise i would just go back and play tetris, for free.

---

Originally by: Kayna Eelai

---

one of my companies customer is one of spains biggest insurance firm. if they have a security breach i am pretty sure they would not dare take the system offline and have all their offices and customers without "system" or without "info". they will take the risk, keep running, their expert team "counter-hacking" and hotfixing, and afterwards they'll take all responsabilities that have to apply. be it leaked information, lost data (which due hot-backup is easy to recover anyways) and fire whoever they have to fire.

so... i don't think it's a "general strategy" to turn off.

---

Originally by: Kayna Eelai

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

1) if you're such a security wonk, how comes your system was not secure enough?
2) I doubt that it's a "much standard practice" to shut down. it depends much on situation, what company you are and what customers you have.

as we've seen in this post, there is many people they don't really give a flying crap about customer service or how many hours they were left without the service they pay for and/or without communication... but other companies might think different and prefer to stay online to give service and have a expert security team hotfixing the problem.

one of my companies customer is one of spains biggest insurance firm. if they have a security breach i am pretty sure they would not dare take the system offline and have all their offices and customers without "system" or without "info". they will take the risk, keep running, their expert team "counter-hacking" and hotfixing, and afterwards they'll take all responsabilities that have to apply. be it leaked information, lost data (which due hot-backup is easy to recover anyways) and fire whoever they have to fire.

so... i don't think it's a "general strategy" to turn off.

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Originally by: Treher

---

To the non-empathetic tards posting "who cares" jabs about people that lost skill time, put down the cheese curls and find a woman.

---

Originally by: Razer Morphis

---

Edited by: Razer Morphis on 20/10/2007 00:05:50
2 cents from someone working in security:

Security breaches are best taken care of... when they are non-existent.

It appears you should focus on more security before **it happens, instead of damage control for the brown nosed.

---

Originally by: CCP Spielmann

---

At 10:25 GMT today we discovered an anomaly in the EVE Online Database indicating a potential exploit. Our policy in such cases is to mobilize a taskforce of internal and external experts to evaluate the situation. At 12:57 that group concluded that our best course of action was to go completely dark while an exhaustive scan of our entire infrastructure was executed.

What we discovered was an indication that one of our databases was being accessed through a security breach. While some may feel that such a drastic reaction was not warranted, it is always our approach to err on the side of caution in order to protect the players.

We of course understand the effect and disruption this has had for our players and apologize for not having been able to explain to the community what was going on. In these cases it can often be counterproductive to containment to give out information while we are in the process of evaluating the scope of the problem.

Our taskforce quickly found the security breach and prevented that from being used. We subsequently found three cases of database actions being performed through the security breach but none of those affected other users than the one doing the actions. We can also confirm that no personal details such as usersÆ credentials or credit card numbers were exposed through this incident. We had to save BOB from losing another Titan.

The servers were brought back online at 22:00 GMT and we will of course continue to monitor the situation closely over the weekend and the following week.

Again we sincerely apologize for this disruption.

Regards,
-J=n H÷r=dal
Chief Operating Officer, CCP

---

Originally by: CCP Wrangler

---

#eve-chaos, #eve and #eve-radio. We also set up a special channel called #eve-announce where people could be sure that what was said was the official news and no rumors.

---

Originally by: Phantom Slave

---

I'm very happy the server is back up, I'm even in doing missions and stuff.

For those people complaining about not getting information, we're paying customers too, and I'm not sure if you noticed but in the EULA it says that there's no guarantee the servers are going to be up 23/7 (scheduled downtime excluded). Deal with it. The rest of us have.

For those that lost skill points, that sucks horribly, and really proves that there needs to be some type of skill queue to ensure that problems like this don't happen again. Unless of course, your second skill finishes during an unexpected downtime, but at least it wasn't totally wasted.

And those that are complaining about the entire server being shut down because of a security breach, you're kidding right? Somebody broke into the database and possibly changed some stuff (not sure if there was stuff changed or not). 'Hotfix it!' I hear you say, and my answer is NO! Shutting down the server was the best damn thing they could have done. Why? Because there was a security breach, and they didn't know how far it had spread. Luckily they caught it and it was just in the server database. It could have spread to billing, or who knows where else. Would you really want the servers to stay online while somebody was peeping through your credit card info, just so you can play a little longer? Hell no.

/end rant

---

Originally by: Selena Rayne

---

A Dev got hit with a keylogger and they used his account to change item spawns to get over 6k officer mods? Is this true or one of the many rumors going around atm?

How the hell does a dev get hit with a keylogger in the first place if this is true

Pretty scary...

---

Originally by: CCP Morpheus

---

Post with your alt.

---

Originally by: Matthew

---

When you're dealing with a security breach on your business-critical systems, you shut it down, and you don't bring any part back up until you're sure you know that bit is safe. Customer communication comes a distant second to protecting critical systems.

---

Originally by: elider

---

Originally by: Ratzap

---

While not impossible that someone outside broke in, most breaches tend to be inside jobs or social engineering attacks on inside people. Someone coming in entirely cold is not the most likely scenario but can always happen (given enough time and determination).
It'll be interesting to see if there's any follow up posts explaining more clearly what/how (doubt it but meh, curious).

Ratzap

---

As linked by anothere post on this thread:

---

Originally by: Madelchai

---

I don't know why people think they're entitled to constant up to the minute responses.
.
.
Having them update you every 15 minutes isn't going to help get it fixed, or make the problem(s) and less severe.

---

Originally by: Kirjava

---

Originally by: Selena Rayne

---

A Dev got hit with a keylogger and they used his account to change item spawns to get over 6k officer mods? Is this true or one of the many rumors going around atm?

How the hell does a dev get hit with a keylogger in the first place if this is true

Pretty scary...

---

Keyloggers can happen to ANYONE.
Dev does not mean invulnerable - many people have lost eve accounts to keyloggers ripping away isk/assets and throwing charecters on ebay.
Let's be glad they didnt decide to do anything major....

---

Originally by: Harkwyth Mist

---

Originally by: Madelchai

---

I don't know why people think they're entitled to constant up to the minute responses.
.
.
Having them update you every 15 minutes isn't going to help get it fixed, or make the problem(s) and less severe.

---

I dont expect an update every 15 minutes. however i DO expect to be told there is a problem and that it being worked on.

---

Originally by: Harkwyth Mist

---

When there are problems with a flight i've booked, i can quickly and easily find out there is a 2-6 hour delay before boarding.
When there are problems with a train i have a ticket for, i can quickly and easily find out the train is running several minutes/hours late.

---

Originally by: Harkwyth Mist

---

Spot the difference with todays events ...

I am not complaining about the servers being down, or about losing training time (because i didnt), but i do have a complaint with the total breakdown of communication from CCP regarding the matter, and in todays multimedia based society i feel that this is totally unnacceptable from a technology orientated company.

---

Originally by: Harkwyth Mist

---

I dont expect an update every 15 minutes. however i DO expect to be told there is a problem and that it being worked on.

---

Originally by: Harkwyth Mist

---

When there are problems with a flight i've booked, i can quickly and easily find out there is a 2-6 hour delay before boarding.
When there are problems with a train i have a ticket for, i can quickly and easily find out the train is running several minutes/hours late.

Spot the difference with todays events ...

---

Originally by: shinsushi

---

So, can't even dignify most of us with a response?

All we are asking for is 1 skill long queue. Not the moon, reimbursement, or someones head on a platter. I guess any type of a response is just too much to ask eh?

---

Originally by: Neth'Rae

---

Edited by: Neth''Rae on 19/10/2007 22:54:12
Ok, but still..

Why are the forums and TQ on the same database?

---

Originally by: Kayna Eelai

---

Originally by: Kirjava

---

Originally by: Selena Rayne

---

A Dev got hit with a keylogger and they used his account to change item spawns to get over 6k officer mods? Is this true or one of the many rumors going around atm?

How the hell does a dev get hit with a keylogger in the first place if this is true

Pretty scary...

---

Keyloggers can happen to ANYONE.
Dev does not mean invulnerable - many people have lost eve accounts to keyloggers ripping away isk/assets and throwing charecters on ebay.
Let's be glad they didnt decide to do anything major....

---

wrong.
if in my company a keylogger would happen, they would first fire the guy in charge of that computers security (me in that case) and secondly fire the guy who browsed websites he should not, while at work.

i really hope (actually i dont hope, i BELIEVE) that the keylogger thing is just a rumour, because i doubt CCP has some1 with so high database access and so stupid to get hit by a keylogger.

---

Originally by: Kayna Eelai

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

1) if you're such a security wonk, how comes your system was not secure enough?
2) I doubt that it's a "much standard practice" to shut down. it depends much on situation, what company you are and what customers you have.

as we've seen in this post, there is many people they don't really give a flying crap about customer service or how many hours they were left without the service they pay for and/or without communication... but other companies might think different and prefer to stay online to give service and have a expert security team hotfixing the problem.

one of my companies customer is one of spains biggest insurance firm. if they have a security breach i am pretty sure they would not dare take the system offline and have all their offices and customers without "system" or without "info". they will take the risk, keep running, their expert team "counter-hacking" and hotfixing, and afterwards they'll take all responsabilities that have to apply. be it leaked information, lost data (which due hot-backup is easy to recover anyways) and fire whoever they have to fire.

so... i don't think it's a "general strategy" to turn off.

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office

---

Originally by: David Hackworth

---

ò If you find yourself in a fair fight, you didn't plan your mission properly.

---

Originally by: Bistot Kid

---

If anyone thinks losing a day of training is any form of issue whatsoever, it's time to step away from your computer for a while and get a breath of fresh air.

It doesn't matter. One little bit.

---

Originally by: TomParad0x

---

You couldnt tell there was a problem?

---

Originally by: TomParad0x

---

So.. your comparing real life items

---

Originally by: TomParad0x

---

What do you think IRC is for?

---

Originally by: Madelchai

---

It's a lot easier to get info on a train or plane than it is to find and fix the cause of a database security issue, and a lot easier to predict a new schedule for it.

---

Quote:

---

Server is back up, sorry

---

Originally by: High Sierra

---

note to self:
dont ever say anything to anyone on the internet about anything ever again.

---

Originally by: Ray Shroff

---

Originally by: Bistot Kid

---

If anyone thinks losing a day of training is any form of issue whatsoever, it's time to step away from your computer for a while and get a breath of fresh air.

It doesn't matter. One little bit.

---

I always see posters like you and laugh on their posts, looks like you are having lots of fun in RL enjoy it, seriously if any one wants to raise the issue about skill training time lost due to incidence like this people like you come and say bla bla you don't have RL , you are whining and stuff but that doesn't mean some one is having less RL enjoyment, or is whining. Someone who lost their skill training time like this and want some thinking from CCP towards this problem and it is not first time this sort of thing happened and nor will be last.
Some people say train for longer skill training time while patch is deployed or some scheduled work is going on, it is perfectly fine and people should set longer skill training time.

But in case of "long" unexpected shutdowns like this CCP should think about it after all it's not our fault for problems like this and if it is discussed there might be some ways to counter it. Honestly i think many CCP guys are wise and they know this is a problem but none is coming forward to answer any post or at least say why it is not possible to even discuss about it.
Now start flaming my guys, but it has to be said.

---

Originally by: David Hackworth

---

ò If you find yourself in a fair fight, you didn't plan your mission properly.

---

Originally by: Rilder

---

Originally by: Dark Shikari

---

Edited by: Dark Shikari on 19/10/2007 22:56:41
First \o/

Thanks for the information... its good to see that CCP is communicating, even as people spread dozens of false rumors claiming doom.

Some rumors I've seen, for laughs:

- Cluster hacked, rollback for obvious reasons.
- Terrorist attack (ranged from bombs on subways to nuclear explosion) on London, TQ servers wrecked
- All sovereignty data reset meaning every single system is back to Sov1 on day 1 including NPC space.
- All wallet data lost, CCP will compensate this by giving everyone one free day of EVE.
- BoB's towers in FAT-6P were coming out of reinforced, CCP crashed the servers to protect them.
- Also, the rollback is for...
-- EVE-TV
-- EVE-O, Dark Shikari is devastated

---

-- All back-ups and data (except for one when Eve came out of beta) lost, everybody starting from scratch.

---

Originally by: Acacia Everto

---

-- Sharkbait had a wild party in the server room and someone tried to use a beer funnel on TQ's SQL server.

---

Originally by: Herc Conley

---

I'd like to make my first post on these forums to say a quick thank you to the game management for the extended downtime. It proved very beneficial for the following reasons:

1. My kids ate three square meals today that I prepared on my day off. I did too come to think of it. I'm not used to such good treatment.

2. I had lots of time to answer some important emails for work. I have an important job. I know they appreciated hearing from me.

3. I cleaned the kitchen, vacuumed all the carpet, took out the trash and even had time to clear up the clogged toilet and a few other home maintenance items. My wife sends her thanks as well.

4. I took the kids outside to play. I saw the sun. It sure seemed brighter than I remembered.

5. Being a recently new player to Eve Online, I got to spend a couple hours on the web reading some tips. It answered so many questions I had about game mechanics - how refreshing.

6. I called my mother back east. I had almost forgotten what she sounded like. She seemed really happy, even wondered aloud if it was her birthday, or mother's day. I thought it was a kind gesture that was long overdue. God bless you mom!

Once again GMs: Thank you!

(thought the place needed a little fresh air)

Herc Conley - the guy in the little ship doing donuts with the windows down outside the CAS spaceport - feel free to say hello and share a brewski.
Damn it's cold in space!

---

Originally by: David Hackworth

---

ò If you find yourself in a fair fight, you didn't plan your mission properly.

---

Originally by: Dred'Pirate Jesus

---

Originally by: Herc Conley

---

I'd like to make my first post on these forums to say a quick thank you to the game management for the extended downtime. It proved very beneficial for the following reasons:

1. My kids ate three square meals today that I prepared on my day off. I did too come to think of it. I'm not used to such good treatment.

2. I had lots of time to answer some important emails for work. I have an important job. I know they appreciated hearing from me.

3. I cleaned the kitchen, vacuumed all the carpet, took out the trash and even had time to clear up the clogged toilet and a few other home maintenance items. My wife sends her thanks as well.

4. I took the kids outside to play. I saw the sun. It sure seemed brighter than I remembered.

5. Being a recently new player to Eve Online, I got to spend a couple hours on the web reading some tips. It answered so many questions I had about game mechanics - how refreshing.

6. I called my mother back east. I had almost forgotten what she sounded like. She seemed really happy, even wondered aloud if it was her birthday, or mother's day. I thought it was a kind gesture that was long overdue. God bless you mom!

Once again GMs: Thank you!

(thought the place needed a little fresh air)

Herc Conley - the guy in the little ship doing donuts with the windows down outside the CAS spaceport - feel free to say hello and share a brewski.
Damn it's cold in space!

---

/Win

---

Originally by: Herc Conley

---

I'd like to make my first post on these forums to say a quick thank you to the game management for the extended downtime. It proved very beneficial for the following reasons:

1. My kids ate three square meals today that I prepared on my day off. I did too come to think of it. I'm not used to such good treatment.

2. I had lots of time to answer some important emails for work. I have an important job. I know they appreciated hearing from me.

3. I cleaned the kitchen, vacuumed all the carpet, took out the trash and even had time to clear up the clogged toilet and a few other home maintenance items. My wife sends her thanks as well.

4. I took the kids outside to play. I saw the sun. It sure seemed brighter than I remembered.

5. Being a recently new player to Eve Online, I got to spend a couple hours on the web reading some tips. It answered so many questions I had about game mechanics - how refreshing.

6. I called my mother back east. I had almost forgotten what she sounded like. She seemed really happy, even wondered aloud if it was her birthday, or mother's day. I thought it was a kind gesture that was long overdue. God bless you mom!

Once again GMs: Thank you!

(thought the place needed a little fresh air)

Herc Conley - the guy in the little ship doing donuts with the windows down outside the CAS spaceport - feel free to say hello and share a brewski.
Damn it's cold in space!

---

Originally by: Akita T

---

SKILL QUEUE

It's on the "to do" list since, like, forever.
Just do it already, ffs. It's not like it's rocket science..

Originally by: CCP Wrangler

---

#eve-chaos, #eve and #eve-radio. We also set up a special channel called #eve-announce where people could be sure that what was said was the official news and no rumors.

---

Originally by: Harkwyth Mist

---

Originally by: TomParad0x

---

What do you think IRC is for?

---

I'm not allowed to use IRC at work

Originally by: Madelchai

---

It's a lot easier to get info on a train or plane than it is to find and fix the cause of a database security issue, and a lot easier to predict a new schedule for it.

---

There was no information at all, hence my complaint.

---

Oh, so let me get this straight, you weren't even playing the game, you were at work.. woha, with the server being down and all I can see how that must have been really frustrating, it's not like you had anything else to do like actually working?

Originally by: KalEl Trask

---

Take for example (3 characters * 9 hours downtime = 18 hours)
When am I going to start getting reimbursed for this additional time?

---

Originally by: KalEl Trask

---

I'm sorry,
This kind of downtime is unexcusable.
I work in an IT support department with over 2000 servers online.
We have to account for any type of downtime over SLA (Service Level Agreement.)
I have 3 characters ($45/month) that I pay for on this service.
The idea that the forum/customer support servers are on the same cluster as the game/databse servers is absolutely/totally absurd!!
This is a complete F@@@up on the companies support line.
There is no reason that the account/credit information is not kept on a completely independant system!!

There should be one way flow with the database/character information.
Is this account active? (Yes/No)
Simple firewall rules should make this happen.

In the environment I support, our client's expect our uptime to be 100% except for the monthly MS sucurity patch reboots!!.
I support the second largest network next to the Department of Defense in the US.)

We are expected to account and compensate for any additional downtime.

This is one of the few MMORGP games that actually schedules consistent downtime/maintenance. Which from an IT standpoint I thins is completely acceptable and applauded.

Unfortunately it seems the downtime/emergency maintenance on this network is an ongoing/recurring issue.

Take for example (3 characters * 9 hours downtime = 18 hours)
When am I going to start getting reimbursed for this additional time?

Time to either start compensating the players for all of the unscheduled downtime or start implementing some real network redundancy!!

---

Originally by: KalEl Trask

---

I'm sorry,
This kind of downtime is unexcusable.
I work in an IT support department with over 2000 servers online.
We have to account for any type of downtime over SLA (Service Level Agreement.)
I have 3 characters ($45/month) that I pay for on this service.
The idea that the forum/customer support servers are on the same cluster as the game/databse servers is absolutely/totally absurd!!
This is a complete F@@@up on the companies support line.
There is no reason that the account/credit information is not kept on a completely independant system!!

There should be one way flow with the database/character information.
Is this account active? (Yes/No)
Simple firewall rules should make this happen.

In the environment I support, our client's expect our uptime to be 100% except for the monthly MS sucurity patch reboots!!.
I support the second largest network next to the Department of Defense in the US.)

We are expected to account and compensate for any additional downtime.

This is one of the few MMORGP games that actually schedules consistent downtime/maintenance. Which from an IT standpoint I thins is completely acceptable and applauded.

Unfortunately it seems the downtime/emergency maintenance on this network is an ongoing/recurring issue.

Take for example (3 characters * 9 hours downtime = 18 hours)
When am I going to start getting reimbursed for this additional time?

Time to either start compensating the players for all of the unscheduled downtime or start implementing some real network redundancy!!

---

Originally by: ChironV

---

This is a game. If you can't cope, go elsewhere.

---

Originally by: Akita T

---

Still, over nine hours of downtime ? Bloody hell...

SKILL QUEUE

It's on the "to do" list since, like, forever.
Just do it already, ffs. It's not like it's rocket science...

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

Originally by: Cadela Fria

---

Who originally fabricated the false chatlog is something entirely different..have no idea who did it.

In other words

Originally by: False

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

This ^^^^ is totally fake. Sharkbait NEVER said any of this.

---

Originally by: KalEl Trask

---

I'm sorry,
This kind of downtime is unexcusable.
I work in an IT support department with over 2000 servers online.
We have to account for any type of downtime over SLA (Service Level Agreement.)
I have 3 characters ($45/month) that I pay for on this service.
The idea that the forum/customer support servers are on the same cluster as the game/databse servers is absolutely/totally absurd!!
This is a complete F@@@up on the companies support line.
There is no reason that the account/credit information is not kept on a completely independant system!!

There should be one way flow with the database/character information.
Is this account active? (Yes/No)
Simple firewall rules should make this happen.

In the environment I support, our client's expect our uptime to be 100% except for the monthly MS sucurity patch reboots!!.
I support the second largest network next to the Department of Defense in the US.)

We are expected to account and compensate for any additional downtime.

This is one of the few MMORGP games that actually schedules consistent downtime/maintenance. Which from an IT standpoint I thins is completely acceptable and applauded.

Unfortunately it seems the downtime/emergency maintenance on this network is an ongoing/recurring issue.

Take for example (3 characters * 9 hours downtime = 18 hours)
When am I going to start getting reimbursed for this additional time?

Time to either start compensating the players for all of the unscheduled downtime or start implementing some real network redundancy!!

---

Originally by: CCP Wrangler[/quote

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

Originally by: KalEl Trask

---

I'm a hugely egocentric douchebag with a superiority complex.

---

Originally by: Skyr

---

I work as a senior programmer at a place that processes medical claims and we are expected to be up 99.99995 of time. To facilitate this, we have monstrous UPS feeding whole building power grid with juice, and generators kicking in when the power outage is out for more than preset amount of seconds.

Our network is triple redundant with all the whistles. Even though the third route would be slow, we would still be up.

NOBODY at our work is permitted to receive mail/browse internet or LOAD any crap on their internal network machines... the email/crap machines are for that.

Should any kind of outage occur for any reasons, we have customer support notifying clients immediately of such incident, and some of them pay as low as $20 per month for all their claims...

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Originally by: Kayna Eelai

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

1) if you're such a security wonk, how comes your system was not secure enough?
2) I doubt that it's a "much standard practice" to shut down. it depends much on situation, what company you are and what customers you have.

as we've seen in this post, there is many people they don't really give a flying crap about customer service or how many hours they were left without the service they pay for and/or without communication... but other companies might think different and prefer to stay online to give service and have a expert security team hotfixing the problem.

one of my companies customer is one of spains biggest insurance firm. if they have a security breach i am pretty sure they would not dare take the system offline and have all their offices and customers without "system" or without "info". they will take the risk, keep running, their expert team "counter-hacking" and hotfixing, and afterwards they'll take all responsabilities that have to apply. be it leaked information, lost data (which due hot-backup is easy to recover anyways) and fire whoever they have to fire.

so... i don't think it's a "general strategy" to turn off.

---

Originally by: Skyr

---

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Originally by: David Hackworth

---

ò If you find yourself in a fair fight, you didn't plan your mission properly.

---

Originally by: Luigi Thirty

---

You know skills don't train when they pull the plug on the SQL server, right? So a queue wouldn't do ****.

---

Originally by: Dred'Pirate Jesus

---

Originally by: Skyr

---

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Why would they want to do that? And more importantly when has a downtime ever exceeded a few hours?

Days? lol..

---

Originally by: Skyr

---

I work as a senior programmer at a place that processes medical claims and we are expected to be up 99.99995 of time. To facilitate this, we have monstrous UPS feeding whole building power grid with juice, and generators kicking in when the power outage is out for more than preset amount of seconds.

Our network is triple redundant with all the whistles. Even though the third route would be slow, we would still be up.

NOBODY at our work is permitted to receive mail/browse internet or LOAD any crap on their internal network machines... the email/crap machines are for that.

Should any kind of outage occur for any reasons, we have customer support notifying clients immediately of such incident, and some of them pay as low as $20 per month for all their claims...

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---