Originally by: Cadela Fria

---

Who originally fabricated the false chatlog is something entirely different..have no idea who did it.

In other words

Originally by: False

---

<&Sharkbait> one of the db admins got hit with keylogger
<&Sharkbait> used an authorized account to shift spawn tables by one row
<&Sharkbait> meta 6 gear was being dropped from standard spawns of the affected npc group
<&Sharkbait> maintenance pruning returned 6539 officer items on one account, so the servers were shut down, forums are integrated and are also shut down
<&Sharkbait> not that long, we have full weekly backups
<&Sharkbait> unfortunately tracing all damage done manually is impossible, we have no tools designed to read transactions when the db itself is skewed
<&Sharkbait> yes, no billing or account status was affected, thats part of the billing dept

---

This ^^^^ is totally fake. Sharkbait NEVER said any of this.

---

Originally by: KalEl Trask

---

I'm sorry,
This kind of downtime is unexcusable.
I work in an IT support department with over 2000 servers online.
We have to account for any type of downtime over SLA (Service Level Agreement.)
I have 3 characters ($45/month) that I pay for on this service.
The idea that the forum/customer support servers are on the same cluster as the game/databse servers is absolutely/totally absurd!!
This is a complete F@@@up on the companies support line.
There is no reason that the account/credit information is not kept on a completely independant system!!

There should be one way flow with the database/character information.
Is this account active? (Yes/No)
Simple firewall rules should make this happen.

In the environment I support, our client's expect our uptime to be 100% except for the monthly MS sucurity patch reboots!!.
I support the second largest network next to the Department of Defense in the US.)

We are expected to account and compensate for any additional downtime.

This is one of the few MMORGP games that actually schedules consistent downtime/maintenance. Which from an IT standpoint I thins is completely acceptable and applauded.

Unfortunately it seems the downtime/emergency maintenance on this network is an ongoing/recurring issue.

Take for example (3 characters * 9 hours downtime = 18 hours)
When am I going to start getting reimbursed for this additional time?

Time to either start compensating the players for all of the unscheduled downtime or start implementing some real network redundancy!!

---

Originally by: CCP Wrangler[/quote

---

I believe there are plans to separate the forums from the server, but that's somewhere in the future. We have however learned from this and will put up a system where we can broadcast messages in similar cases as this. We of course hope we wont have to do that for quite some time, if ever.

---

Originally by: KalEl Trask

---

I'm a hugely egocentric douchebag with a superiority complex.

---

Originally by: Skyr

---

I work as a senior programmer at a place that processes medical claims and we are expected to be up 99.99995 of time. To facilitate this, we have monstrous UPS feeding whole building power grid with juice, and generators kicking in when the power outage is out for more than preset amount of seconds.

Our network is triple redundant with all the whistles. Even though the third route would be slow, we would still be up.

NOBODY at our work is permitted to receive mail/browse internet or LOAD any crap on their internal network machines... the email/crap machines are for that.

Should any kind of outage occur for any reasons, we have customer support notifying clients immediately of such incident, and some of them pay as low as $20 per month for all their claims...

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Originally by: Kayna Eelai

---

Originally by: ISD Rauth Kivaro

---

As a security wonk in my real life:

In a security breach situation, it's pretty much standard practice to shut down and say absolutely nothing until you're totally certain you have the situation in hand. Any leak of information can potentially magnify the problem.

It may seem draconian but when you have as much at stake as there is here, it's logical.

---

1) if you're such a security wonk, how comes your system was not secure enough?
2) I doubt that it's a "much standard practice" to shut down. it depends much on situation, what company you are and what customers you have.

as we've seen in this post, there is many people they don't really give a flying crap about customer service or how many hours they were left without the service they pay for and/or without communication... but other companies might think different and prefer to stay online to give service and have a expert security team hotfixing the problem.

one of my companies customer is one of spains biggest insurance firm. if they have a security breach i am pretty sure they would not dare take the system offline and have all their offices and customers without "system" or without "info". they will take the risk, keep running, their expert team "counter-hacking" and hotfixing, and afterwards they'll take all responsabilities that have to apply. be it leaked information, lost data (which due hot-backup is easy to recover anyways) and fire whoever they have to fire.

so... i don't think it's a "general strategy" to turn off.

---

Originally by: Skyr

---

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Originally by: David Hackworth

---

ò If you find yourself in a fair fight, you didn't plan your mission properly.

---

Originally by: Luigi Thirty

---

You know skills don't train when they pull the plug on the SQL server, right? So a queue wouldn't do ****.

---

Originally by: Dred'Pirate Jesus

---

Originally by: Skyr

---

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Why would they want to do that? And more importantly when has a downtime ever exceeded a few hours?

Days? lol..

---

Originally by: Skyr

---

I work as a senior programmer at a place that processes medical claims and we are expected to be up 99.99995 of time. To facilitate this, we have monstrous UPS feeding whole building power grid with juice, and generators kicking in when the power outage is out for more than preset amount of seconds.

Our network is triple redundant with all the whistles. Even though the third route would be slow, we would still be up.

NOBODY at our work is permitted to receive mail/browse internet or LOAD any crap on their internal network machines... the email/crap machines are for that.

Should any kind of outage occur for any reasons, we have customer support notifying clients immediately of such incident, and some of them pay as low as $20 per month for all their claims...

it seems like CCP's plan is to keep us in permanent DT stage with days without any service and ANY communication whatsoever.

Thanks.

---

Originally by: Nhilist

---

Originally by: elider

---

CCP you want me to belive that:
-An EVE-Online player connects from his little laptop and hacks your firewall
-Then he magically knows the internal structure of your networks and finds the database server
-Then because he is such a genius , he is able to hack the security of the database server too...
- Then he is able to figure out in some minutes the structure of such a complex database so he finds his wallet entry and puts there some ISK

Just LOL!!! I bet most of your programmers are not able to do that from his desk in CCP office and you want us to belive someone did that from home without any knowledge of internal network arhitecture or database structure?

One that really belives this .....must have seen too many bad Hollywood movies

---

shut up.

select * from sys.objects where type = N'U' is really hard. . .

---

Originally by: XoPhyte

---

Originally by: Nhilist

---

Originally by: elider

---

-baseless assumptions about hacking or computer security-

---

shut up.

select * from sys.objects where type = N'U' is really hard. . .

---

Bastard, beat me to it!

Oh, and the fact that because these forums connect to the same SQL server & database that it *could* have been a SQL injection in which you dont need to know the internal network at all, only the database structure which has already been published, and you wouldn't have to get past the firewall at all, since the firewall is already allowing port 80 (http) access to these forums, and these forums have access to the backend SQL servers.

Though I would seriously doubt (and hope) that CCP is using an ID for forums with only DB_READER access to a few tables in the database (though again I would suspect its a different database).

Regardless the standard port for SQL is 1433, it's not very difficult to scan it down on the network if in fact you are connected via some method to the internal network. Getting a userID with DB_WRITER access though would be a bit more complex, but again I suspect that the hacker would either utilize an existing piece of code which has RW access to the database, or came across it in a piece of code (which is a scary thought).

Anyway, there are a billion methods in which to hack a server, but judging by your complete lack of comprehension elider, I wouldn't expect you to understand any of them.

---

Originally by: Vorok

---

Originally by: XoPhyte

---

Originally by: Nhilist

---

Originally by: elider

---

-baseless assumptions about hacking or computer security-

---

shut up.

select * from sys.objects where type = N'U' is really hard. . .

---

Bastard, beat me to it!

Oh, and the fact that because these forums connect to the same SQL server & database that it *could* have been a SQL injection in which you dont need to know the internal network at all, only the database structure which has already been published, and you wouldn't have to get past the firewall at all, since the firewall is already allowing port 80 (http) access to these forums, and these forums have access to the backend SQL servers.

Though I would seriously doubt (and hope) that CCP is using an ID for forums with only DB_READER access to a few tables in the database (though again I would suspect its a different database).

Regardless the standard port for SQL is 1433, it's not very difficult to scan it down on the network if in fact you are connected via some method to the internal network. Getting a userID with DB_WRITER access though would be a bit more complex, but again I suspect that the hacker would either utilize an existing piece of code which has RW access to the database, or came across it in a piece of code (which is a scary thought).

Anyway, there are a billion methods in which to hack a server, but judging by your complete lack of comprehension elider, I wouldn't expect you to understand any of them.

---

I feel that the following is an appropriate contribution to this discussion: http://xkcd.com/327/

---

Originally by: Kayna Eelai

---

i am a paying customer and i can't believe you had us sitting in the complete dark FOR HOURS with zero information. we had t oaccess 3rd party places to get (unconfirmed) bits and pieces of information. things only started to light up a bit when everyone went to eve radio to get some more info.

very bad towards your customers. worse than blizzard, and that means a lot.

---

Originally by: Neth'Rae

---

Edited by: Neth''Rae on 19/10/2007 22:54:12
Ok, but still..

Why are the forums and TQ on the same database?

---