Pages: 1 2 3 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 6 post(s) |
|
CCP Fallout
|
Posted - 2010.05.13 17:24:00 -
[1]
Friday, May 7th, was an interesting and rough day for both players and CCP. What happened, and how can we prevent future occurances? CCP Red Button returns with a post mortem of the events in his newest dev blog.
Fallout Associate Community Manager CCP Hf, EVE Online Contact us |
|
gerash
Gallente Q3 - TnR
|
Posted - 2010.05.13 17:31:00 -
[2]
I was wondering what policies ccp has in place for recovering from hardware failures specifically a loss of the database server. Is there any coherent backup strategy in place?
|
Agent Unknown
Caldari
|
Posted - 2010.05.13 17:41:00 -
[3]
Insightful blog. Unfortunately things can go haywire, but at least it didn't affect all the items in EVE...that would be even worse. By the way, this is my signature.
TeamSpeak For EVE - API-controlled TeamSpeak 3 Access!
|
GateScout
|
Posted - 2010.05.13 17:56:00 -
[4]
Thanks for the informative dev blog. That explain a lot. More of that please. Thanks!
|
|
CCP Red Button
|
Posted - 2010.05.13 17:57:00 -
[5]
To answer your question Gerash the Tranquility main database which is at the heart of EVE is clustered and fully redundant in every way. If needed we can recover the database up to the minute from backup but the problem is the time it currently takes. So what we aim to do now is being able to instantly switch to a time-delayed mirror in addition to drastically reducing backup restoration times. ------
Hmmm... wonder where this cable goes 0_o |
|
Borgh Brainbasher
Path of Now and Forever
|
Posted - 2010.05.13 17:58:00 -
[6]
so, CCp Red Button, is that the guy who gets hit over the head when the server goes haywire? --- Warning: You are on the pvp only server. |
Darth Vapour
|
Posted - 2010.05.13 17:59:00 -
[7]
Quote: What all the scrutiny and testing of this script had failed to take into consideration was the huge difference in transaction volume on Tranquility vs. Singularity.
That's kind of amateurish. I hope this was a one-off case and some awareness exists in CCP land that your live and test systems are not comparable in how they behave.
|
Luke S
Zeta Corp.
|
Posted - 2010.05.13 18:07:00 -
[8]
Thanks Red. but its still easier to blame Tuxford. ---
|
Rene Sauntier
Gallente Descendants of Hermes
|
Posted - 2010.05.13 18:27:00 -
[9]
nice blog.poast, it should stop some people dribbling.
+1 for informative bug fixes "What I cannot create, I do not understand." |
Jamier Legov
Minmatar Blue Republic
|
Posted - 2010.05.13 18:30:00 -
[10]
Quote: The main lesson learned last Friday, however, is that we need to step up our game in terms of speedy restoration of backups and options for recovery.
That's like closing the barn door after the horse has left....
The main lesson learned should be that testing needs to take into account the differences between Singularity and Tranquility. When your testing, you need to test with the actual volume of transactions on the production enviroment. Load testing isn't just a good idea, or a software development best practice, it's the right way to prevent problems in the first place.
|
|
Paddlefoot Aeon
SiN. Corp Daisho Syndicate
|
Posted - 2010.05.13 18:38:00 -
[11]
I remember reading that, in the next patch, there will be the ability to run "ghost clients" for developers, to simulate in the DEV environment the high-load conditions typically seen in the production (Tranq.) DB.
Will running these clients help avoid these high-volume-only bugs from making it "live"?
|
The Pricer
|
Posted - 2010.05.13 18:55:00 -
[12]
Originally by: Luke S Thanks Red. but its still easier to blame Tuxford.
ONE DOES NOT SIMPLY PRESS CCP RED BUTTON!!!!
|
Cinori Aluben
Minmatar Gladiators of Rage
|
Posted - 2010.05.13 18:56:00 -
[13]
Wow, great transparency in helping us understand what's going on. Keep it up. And keep up the work to avoid this in future.
I will give props to the GMs in their petition responses for this one. I lost a sacrilege from a buy order on this one, and they gave it back within like 2 days. Good job guys.
Cinori Aluben CSM5 2010Fix the Little Things First!
---
Cinori Aluben -- CSM 2010!! "Fix the Little Things First!" http://www.littlethingsfirst.com |
Gustovness
EntroPrelatial Vanguard EntroPraetorian Aegis
|
Posted - 2010.05.13 18:57:00 -
[14]
I applaud CCP for putting out this dev blog. I think I speak for the majority of your customers that we appreciate your filling us in on these catastrophic disasters and keeping us in the loop. I didn't have any problems myself with this episode, but I like hearing that you guys are taking the time to keep us all informed as to what's going on.
thumbs up
|
Mynxee
Noir. Noir. Mercenary Group
|
Posted - 2010.05.13 19:04:00 -
[15]
Thanks for an informative "behind the scenes" explanation...much appreciated!
Life In Low Sec |
Nierna
|
Posted - 2010.05.13 19:09:00 -
[16]
it is these kinds of devblogs that make me love the CCP developers. other games would simply say "we had a problem and we fixed it" youguys actualy explain the problem and fix implemented. please do keep up this level of transperaty.
|
|
Chribba
Otherworld Enterprises Otherworld Empire
|
Posted - 2010.05.13 19:22:00 -
[17]
Interesting read. Keep up the good work.
/c
Secure 3rd party service | my in-game channel 'Holy Veldspar' |
|
Nareg Maxence
Gallente
|
Posted - 2010.05.13 19:26:00 -
[18]
Originally by: Jamier Legov
Quote: The main lesson learned last Friday, however, is that we need to step up our game in terms of speedy restoration of backups and options for recovery.
That's like closing the barn door after the horse has left....
The main lesson learned should be that testing needs to take into account the differences between Singularity and Tranquility. When your testing, you need to test with the actual volume of transactions on the production enviroment. Load testing isn't just a good idea, or a software development best practice, it's the right way to prevent problems in the first place.
I am qurious as to how exactly you would arrange such a test.
|
DmitryEKT
Atlas Alliance
|
Posted - 2010.05.13 19:45:00 -
[19]
Rollback would have been cool tho :o
|
Haven Wind
|
Posted - 2010.05.13 19:58:00 -
[20]
Love CCP, Classy response in an industry where often there is no response or communication with the player base at all over these types of issues.
|
|
Grimpak
Gallente Noir. Noir. Mercenary Group
|
Posted - 2010.05.13 20:13:00 -
[21]
Originally by: DmitryEKT Rollback would have been cool tho :o
DON'T EVEN THINK ABOUT IT!
you can't even fathom what the **** happened when the last rollback happened. ---
Quote: The more I know about humans, the more I love animals.
ain't that right. |
Mihara Shiharu
|
Posted - 2010.05.13 21:07:00 -
[22]
Time to get a better database really :P
|
Derus Grobb
RipStar Mining Industries United Front Alliance
|
Posted - 2010.05.13 21:08:00 -
[23]
Interesting button.. i mean devblog ---
|
T'Amber
www.shipsofeve.com
|
Posted - 2010.05.13 21:21:00 -
[24]
Edited by: T''Amber on 13/05/2010 21:23:36
Ah.. some of the votes I bought off the market didn't show up that day and I was wondering what happened. thanks for the explanation
-T'amber
POLITICS:SIMULATORÖ
|
Sul Trewtis
ANZAC ALLIANCE IT Alliance
|
Posted - 2010.05.13 21:28:00 -
[25]
Wait wait wait.....
Your telling us the logs actually showed something?
*pods himself in shock*
And unfortunately **** like this happens no matter what procedures you have in place. All you can really do is be professional and make sure it doesn't happen often or ever repeat. -------------------------------------------------- Civil Disobedience is still Disobedience The most damage is done by those that know how and not why |
Lagruna Zegata
|
Posted - 2010.05.13 21:42:00 -
[26]
Always unfortunate to see these things happen to such a terrific game, especially to divert CCP's attention right before a much-awaited expansion. Kudos for explaining everything as thoroughly as possible.
Some semi-technical side questions: How come this new area of the junkyard needed to be cleaned? And why was a modified script necessary for this area?
~LZ
|
|
CCP Red Button
|
Posted - 2010.05.13 21:55:00 -
[27]
Interresting question Lagruna. To tell you the truth I am not sure why this section of the junkyard needed cleaning at this point. Each section of the junkyard is used for a different set of things as far as I know and thus might fill up at different rates so that might explain why. I will try to find out tomorrow. The reason for the modification was to optimize performance.
------
Hmmm... wonder where this cable goes 0_o |
|
Lost Hamster
Hamster Holding Corp
|
Posted - 2010.05.13 21:56:00 -
[28]
It's good to read about the problem. Many of us thought that there was a rollback, as some of the items where missing after the long downtime.
|
Laendra
Universalis Imperium
|
Posted - 2010.05.13 22:12:00 -
[29]
So what do we look for to see if we have been affected by this? At what point is it "too late" to do anything about it? -------------------
|
Nadarius Chrome
Celestial Horizon Corp.
|
Posted - 2010.05.13 22:51:00 -
[30]
Originally by: Laendra So what do we look for to see if we have been affected by this? At what point is it "too late" to do anything about it?
I'm in a similar situation. I have quite a few market orders, but no formal way I track them, and dread to think if there's a chance one or several of mine were affected. I suspect the chance is low but still that nagging doubt...
And I don't want to lodge a petition merely to ask "Hey, could I have been affected?" because I'm sure the GMs handling that queue are plenty busy with petitions from people who did lose items as well as petitions from people who didn't but are trying to scam free stuff. |
|
Tamasan
Gallente BLACK LABEL INC Malum Exuro
|
Posted - 2010.05.14 00:07:00 -
[31]
It has been my experience that those *profanity* stupid one line coding errors are the hardest to find and fix.
Kudos to CCP for finding the problem quickly, and also making a tough choice that was better for the players (avoiding rollback), even if more time consuming for all the GMs having to deal with the petitions.
|
Abdiel Kavash
Caldari Paladin Order Fidelas Constans
|
Posted - 2010.05.14 00:54:00 -
[32]
Originally by: Lagruna Zegata Always unfortunate to see these things happen to such a terrific game, especially to divert CCP's attention right before a much-awaited expansion. Kudos for explaining everything as thoroughly as possible.
Some semi-technical side questions: How come this new area of the junkyard needed to be cleaned? And why was a modified script necessary for this area?
~LZ
I am guessing that this "junkyard" area has separate divisions for items, ships, characters, and whatever else ends up there. One reason I see for this setup is being able to find stuff easier in case you need to restore something. And as you might imagine, the pool of deleted items fills up much faster than, say, deleted characters. But over time, even the second one starts taking up too much space and IDs and needs to be purged.
Keep in mind that this is only my best guess, the actual reason might or might not be similar to this.
|
Louis deGuerre
Gallente Amicus Morte Shock an Awe
|
Posted - 2010.05.14 01:02:00 -
[33]
I remember two years ago my company had a 'black week' and were forced to do a rollback due to dramatic multiple hardware failure. We were at that time not ready for a catastrophe of such magnitude and it took us almost three days. For an on-line service, that is high drama indeed. So I fully understand and support CCP's decision to not do a rollback.
And I'm just gonna say it, this will go down in eve history as 'Black Friday'
Love these blogs ! Keep em coming ! Sol: A microwarp drive? In a battleship? Are you insane? They arenÆt built for this! Clear Skies - The Movie
|
Tres Farmer
Gallente Federation Intelligence Service
|
Posted - 2010.05.14 01:46:00 -
[34]
Hm.. ID's which piled up since now and never got reasigned.. what could this be - T2 BPO Id's?
Anyways.. if it affected market orders the most I would guess this part of the junkyard gets wiped the most (short behind the NPC junkyard Id's I guess) and reused those new Id's first.
Very nice Blog.. want to read more from the Red Button! Is there by any chance a 'CCP Run for yer lives' or a 'CCP Were Screwed' on board?
|
Ms Michigan
|
Posted - 2010.05.14 02:12:00 -
[35]
So.... Not to be the "Glumy Gus" here but this whole answer seems good but not good enough...
Fact: Items went missing. Fact: You are leaving it up to players to NOTICE what went missing (if one or more of the 114K+ items was one of theirs). Fact: No ONE in EVE, sub the noob with only a mining laser and ship to his name, remembers all of their items in and around all of New Eden.
So.... my question is: How do I know if I am affected? and What is CCP doing about it besides telling us (which I am not knocking) but given the afore mentioned - seems a neglect? I mean - I know I would notice if my Titan went missing, but what about players that have so many items and log in infrequently. How do I know if that faction hardner I thought I had in Jita is gone or not?
|
Nadarius Chrome
Celestial Horizon Corp.
|
Posted - 2010.05.14 03:44:00 -
[36]
As far as I understand, only items that underwent a transaction while there were no IDs available were affected. Items just sitting in your hangar unmolested would not have disappeared.
EG someone sold you something by buy order, or you delivered a manufacturing run, or stacked some items. |
Diomedes Calypso
|
Posted - 2010.05.14 03:57:00 -
[37]
Not trying to troll here...I'm just a curious guy that likes to think of things from different angles and fill holes in my general models of different things I carry around in my mind.
Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ? (I can see problems rolling back medical records or stock trades)
Sure, like real world stock trades, people would give up any gains (or recoup losses) that happened during the intervening time but that seems fairly workable in a game. Its like taking a goal of the scoreboard if had rolled onto the field during the play before it.
There would be winners and losers from the event, but there are also winners and losers if you don't roll back and given the choice to corrupted information and erased action the erased action seems most consitent with rules of sport ?
But there must be some techical issue that I don't get ? Is it that a roll back can't truly restore the past state ?
|
cpu939
Gallente Volatile Nature Systematic-Chaos
|
Posted - 2010.05.14 04:18:00 -
[38]
how many junkyards are there? how many items are in each junkyards? also what is the daily recycled average i.e. number of items cleared of the list? how many of the dev team did it take to fix this problem? i bet the mayday call went out for all the devs to come quick (sry bad pun) 0101011 001101111 011011000 110000101110100 01101001011011000 1100101001000000 1001110011000010 11101000111010101 11001001100101
|
Callipygian Provocateur
BIG Majesta Empire
|
Posted - 2010.05.14 06:21:00 -
[39]
Diomedes Calypso, as Red Button mentioned, a rollback would have taken a long time, probably hours considering it sounded like it would involve restoring a backup copy of the entire EVE database, which, I suspect, is on the order of TBs or even 10s of TB. Or, the possible alternative of 'walking backwards' through the transaction logs isn't something CCP has software in place to do, and could potentially cause further complications.
Database administration isn't clean, easy or sexy. Writing scripts to interact with a busy, concurrent DB isn't simple stuff. It sounds like the problem in the script was one of timing, where item IDs were flagged as available before they truly were from the DB's perspective, but only by the tiniest fraction of a second, which is why the error didn't occur on Singularity.
Anyway, thanks a ton for the frank, honest communication. I think CCP's transparency and openness with the players is a huge part of why this community is so supportive.
|
|
CCP Red Button
|
Posted - 2010.05.14 09:39:00 -
[40]
Edited by: CCP Red Button on 14/05/2010 09:40:52 Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?
Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint. ------
Hmmm... wonder where this cable goes 0_o |
|
|
123GuessWho
|
Posted - 2010.05.14 09:48:00 -
[41]
Originally by: CCP Red Button Edited by: CCP Red Button on 14/05/2010 09:40:52 Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?
Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint.
I remember this happening in 2008 during a MAX campaign, all titans, capitals and pos' where back up after a server crash. So that must have been a roll back
|
wizard87
|
Posted - 2010.05.14 09:51:00 -
[42]
Disaster Recovery, Roll-backs, Backups, Failovers.
Those words bring back horrible memories of DR tests in a former life. As for Eve its about time you realised the disaster occurred about 3 or 4 years ago around the time you removed Escrow....since then its been one long FailOveur - time for a roll-back.
|
|
CCP Red Button
|
Posted - 2010.05.14 09:52:00 -
[43]
Originally by: cpu939 how many junkyards are there? how many items are in each junkyards? also what is the daily recycled average i.e. number of items cleared of the list? how many of the dev team did it take to fix this problem? i bet the mayday call went out for all the devs to come quick (sry bad pun)
So many questions and so few answers
To tell you the truth I don't really know how many there are. There are several, I know that much, and the number of items in each varies depending on what it contains. The recycled average also varies depending on the activity levels in game and the frequency of cleanups.
There were quite a few people working on fixing this problem last Friday, not just DBAs and programmers but individuals from almost every part of the company, customer support, community, operations, engineering etc. Essentially when EVE has a problem, the whole of CCP drops whatever they are doing and comes running. ------
Hmmm... wonder where this cable goes 0_o |
|
Crudamus
hirr Morsus Mihi
|
Posted - 2010.05.14 09:54:00 -
[44]
I have 10's of thousands of items. So are you saying that it is possible for me to be missing something and would not even know it because I didn't bother to look over my assets?. Like a ship, ammo, BM, mod or even my chars on an account I don't have active?
|
Cebraio
|
Posted - 2010.05.14 10:38:00 -
[45]
Originally by: Crudamus I have 10's of thousands of items. So are you saying that it is possible for me to be missing something and would not even know it because I didn't bother to look over my assets?. Like a ship, ammo, BM, mod or even my chars on an account I don't have active?
I was going to ask exactly that. Which item categories were affected? How are we supposed to find out if we lost something valuable?
|
|
CCP Red Button
|
Posted - 2010.05.14 10:49:00 -
[46]
Crudamus, no existing items were affected by this error. This only affected a small portion of new items being created during the 60 minute time period the script was running. That includes market orders being completed, items stacked or unstacked, new characters created etc. Most of these we were able to fix during Fridays downtime and we are still working on finding and fixing the rest. So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate. ------
Hmmm... wonder where this cable goes 0_o |
|
clixoras
|
Posted - 2010.05.14 11:00:00 -
[47]
Originally by: CCP Red Button and we are still working on finding and fixing the rest. So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate.
Are you saying that you are investigating ALL items that are missing? or ONLY the ones that got petitioned? Like some others i don't have a clue what could be missing (with the variables you gave ofcourse). Could be a few mil but also 1b.
|
Hawk TT
Caldari Bulgarian Experienced Crackers Circle-Of-Two
|
Posted - 2010.05.14 11:13:00 -
[48]
Check out "Datacore Continuous Data Protection" ;-)
http://www.datacore.com/products/prod_Traveller.asp
Instant "time-machine" roll-backs on storage (SAN) level + other extras. Such solution would require virtualizing your RAMSANs, but it has also multiple other benefits. With the current "physical/fixed" SAN architecture you are pretty much "handcuffed"... ___________________________________ Science & Diplomacy Manager @ BECKS Circle-of-Two |
Shasz
Detox.
|
Posted - 2010.05.14 11:41:00 -
[49]
I love reading the details like this, thank you very much CCP for keeping us in the loop.
If I understand correctly, the reason CCP can't just search for and correct every issue is because much of it requires research into the claim and a judgement call.
Consider this example: You, the unsuspecting pilot, buy a new Dominix in Jita. That new ship needs an item id and grabs the next available one. Somewhere else a pilot took delivery of his freshly produced Navy Raven from the LP store, and it too needs an item id. Due to the bug in script, both pilots got the same item id for their ship. I'm guessing the last one to log out and/or save that ship id is going to be the one that "wins" and keeps his ship the next time he logs in. The database can't save both ships under the same id, it can only overwrite one with the other. Now that case is easily checked and verified - there will be a record of the purchase of both ships and the missing ship can be safely re-created. This could even be automated.
But what about creating a new item id by unstacking items? Say you had a stack of 3 unassembled Dominixes instead, and you peeled one off the stack to fit it up, and your item id collided with another new ship. There probably wouldn't be a record of you simply unstacking a ship, and someone would have to search a backup of the database to verify that you did indeed own 3 Domis in the recent past, and that you hadn't lost one in combat during the bug timeframe. Writing a script to verify all that just isn't reasonable, and each case has to be petitioned as well because there's no way to automatically determine something is missing.
HTH ___________________________________
|
Sable Moran
Gallente Moran Light Industries
|
Posted - 2010.05.14 11:43:00 -
[50]
To all the whiners..
CCP t0rfifrans has explained quite clearly how to determine if you're affected by this issue or not in this news item.
Read it, apply brainpower and.... profit!?!?? ----- Sable's Ammo Shop at Alentene 5, Moon 4, Duvolle Laboratories Factory. Ammo at affordable prices. |
|
Vint Rotach
|
Posted - 2010.05.14 14:27:00 -
[51]
+1
Open and honest.
Wish the 'collection of individuals' I work for could work with the same idea.
|
Sleeper King
|
Posted - 2010.05.14 18:58:00 -
[52]
Edited by: Sleeper King on 14/05/2010 19:00:50
Originally by: Hawk TT
Check out "Datacore Continuous Data Protection" ;-)
http://www.datacore.com/products/prod_Traveller.asp
Instant "time-machine" roll-backs on storage (SAN) level + other extras. Such solution would require virtualizing your RAMSANs, but it has also multiple other benefits. With the current "physical/fixed" SAN architecture you are pretty much "handcuffed"...
Wait... Don't you have to be in the IT Alliance to make suggestions like this? (joke) How do I drop in an application to the CCP Alliance in Jove Space pls?
Also, thanks for making the best decision you could in the heat of the moment that ultimately benefited the largest number of players.
Edit - Id also like to seem some statistics on the number of fail scam petitions for items that were not lost.
I lost a Eurbus BPO btw... |
Meno Theaetetus
Wildly Inappropriate Wildly Inappropriate.
|
Posted - 2010.05.14 19:43:00 -
[53]
Originally by: CCP Red Button Edited by: CCP Red Button on 14/05/2010 09:40:52 Edited by: CCP Red Button on 14/05/2010 09:40:23
Originally by: Diomedes Calypso Could someone give me a few sentences on why rol-backs would be a nightmare, especially in a game ?
Fair question Diomedes. The reason why we consider rollbacks an absolute last resort thing is that by doing so, even by a short period, you are loosing some of EVE History. Imagine a big battle being fought and then suddenly it never happened, a successful mission completed and then it never was, that ingenious market trade.. etc. So we will essentially explore every other option and turn every stone before deciding to rollback, even if that would in some cases be the easiest option from a technical standpoint.
Just a quick heads up to ccp and empire people who don't seem venture out of >=0.5, small rollbacks occur nearly weekly when nodes crash, one minute no ship, the next .. oh I'm back here in my ship again + my insurance.
|
Antihrist Pripravnik
4S Corporation Morsus Mihi
|
Posted - 2010.05.14 20:47:00 -
[54]
+1 for the great devblog |
Alain Kinsella
Minmatar
|
Posted - 2010.05.14 21:33:00 -
[55]
Thanks for the update. As a Systems Admin (and backup admin, and a DR/BK 'point man') I sympathize completely.
Have you had a chance to look at faster replacements to RamSan? I hear FusionIO is doing some rather interesting things now, for 1/10th the price.
[Yes, possible heresy, but we have RamSan units at work and so far they've only hit their best potential with high-end SPARC servers driving them.]
--A_K
|
arjun
Battlestars Wildly Inappropriate.
|
Posted - 2010.05.15 06:51:00 -
[56]
i petitioned for the probable loss of 1,9 b through the market. no answer yet. is there still a que the gms work on or is this thing "ad acta" ?
|
Nadarius Chrome
Celestial Horizon Corp.
|
Posted - 2010.05.16 22:17:00 -
[57]
Originally by: CCP Red Button So if you were active and logged on in that time period and think this might have affected you I would encourage you to send in a petition and the GM's will investigate.
So what you're saying is, anyone who cannot definitively rule out any loss whatsoever from this should lodge a petition?
Can a GM skim through my transactions and stuff and see easily what, if anything, disappeared?
If so, incoming petition. |
Jennifer Christage
Token Industries
|
Posted - 2010.05.18 13:00:00 -
[58]
So, my petition sent minutes after the problems started still hasn't been fully resolved and no reply from a GM for almost a week. What exactly am I expected to do for you to resolve this problem?
|
Rip Minner
Gallente Balthizar Drako Inc.
|
Posted - 2010.05.20 07:58:00 -
[59]
Edited by: Rip Minner on 20/05/2010 08:01:11 haha I find it very funny that the guy that gets called in for the worst is Called Red Button. Perfect name for the job :) Is it a rock? Point a Lazer at it and profit. Is it a ship? Point a Lazer at it and profit. I dont realy see any differnces here. |
Blitz0r
Minmatar M. Corp Daisho Syndicate
|
Posted - 2010.05.20 13:29:00 -
[60]
Thank you for a wonderful report!
Good luck chasing the errors and keep up the good work
M.Corp BPC Packages |
|
|
|
|
Pages: 1 2 3 :: [one page] |