Pages: 1 2 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 1 post(s) |
|
CCP Fallout
|
Posted - 2009.04.08 13:15:00 -
[1]
Very few capsuleers were not affected by the severe lag that haunted Jita post-Apocrypha launch. So it was with much relief when our developers resolved the Jita issues. But how did they work this miracle? In his latest blog, CCP Atlas gives us the lowdown on not only how they searched and destroyed the issue, but how they fixed it... live.
Fallout Associate Community Manager CCP Hf, EVE Online Contact us |
|
|
Chribba
Otherworld Enterprises Otherworld Empire
|
Posted - 2009.04.08 13:20:00 -
[2]
I thought it was 'Real men Hull tank'?
- |
|
|
Chribba
Otherworld Enterprises Otherworld Empire
|
Posted - 2009.04.08 13:24:00 -
[3]
And nice to see live code injects done properly
- |
|
Jameroz
Echoes of Space
|
Posted - 2009.04.08 13:25:00 -
[4]
Now ... who wrote that one line?
|
hfo ohforf
|
Posted - 2009.04.08 13:27:00 -
[5]
cool! :) (Havent been in jita for some time now, but i'm looking forward to seeing how the Heisenberg compensators are doing)
|
Ix Forres
Caldari Vanguard Frontiers Sc0rched Earth
|
Posted - 2009.04.08 13:30:00 -
[6]
Are there any estimates on how this could affect non-Jita nodes in terms of responsiveness and reliability? -- Ix Forres EVE Application Developer ISKsense | EVE Metrics (NEW) | I Tweet |
LaVista Vista
Conservative Shenanigans Party
|
Posted - 2009.04.08 13:30:00 -
[7]
Awesome stuff.
Can you confirm that the developer who wrote that piece of code, had to buy beer for everybody? That seems only logical.
|
Ralitge boyter
Minmatar Scalar Federation
|
Posted - 2009.04.08 14:18:00 -
[8]
So hang on, what was the problem in that line of code? And what caused that line of code to be there in the first place, was it something that has always been there or are we talking about a minor mistake during the development of a new bit of code leading to major issues?
Did you just add a sleep(10) to the end of every cycle or is it a silly loop that took to long or what whas the actual reason Jita and other systems where so very slow... ------------------------------------------- Should you disagree with me, well I guess that is because I disagree with you. If you have a problem with that please feel free not to tell me. |
Bambi
Existentialist Collective
|
Posted - 2009.04.08 14:34:00 -
[9]
Originally by: Chribba I thought it was 'Real men Hull tank'?
Real (wo)men pod tank....
EVE is dead, long live EVE!
|
Eleana Tomelac
Gallente Eclats de verre
|
Posted - 2009.04.08 14:49:00 -
[10]
Good job on finding out, I am just wondering if that exact issue was causing this :
In dodixie, with over 400 people, my ship's capacitor was recharging at a normal rate (when looking at my watch), while the module activation (for NPC too, hopefully) were longer to work, making my Sleipnir's capacitor godlike, allowing the use of Hail M and permanent LSB II with AB and invulns on, while the cap never came under 70% (it should just collapse under all modules activated with hail lowering the recharge time)?
Well, never had such good 'capacitor tank'! -- Pocket drone carriers (tm) enthousiast ! |
|
Attrezzo Pox
Amarr Navy of Xoc Wildly Inappropriate.
|
Posted - 2009.04.08 15:29:00 -
[11]
I'm not a woman, but if I were I would think that this blog was hot. Sexy even.
If there were women directly involved then I think they are hot.
In a programmer's world that was as good as a fresh Hustler or Playboy article.
Score CCP! YEAH BABY! a real man (or woman). score. *-------------------------* PoX IS Eve!!! BOOM!!! |
Marcus Arelios
Caldari O.W.N. Corp OWN Alliance
|
Posted - 2009.04.08 15:31:00 -
[12]
Nice job guys, finding one line bugs is not an easy task, let alone when your dealing with server cluster/network issues.
|
Imhothar Xarodit
Minmatar Wolverine Solutions Dead Mans Hand
|
Posted - 2009.04.08 16:11:00 -
[13]
Edited by: Imhothar Xarodit on 08/04/2009 16:13:52 SVN BLAAAAAMMMMMMMEEEEE (in our company it has become a tradition to loudly shout BLAAAAAMMMMMMMEEEEE through the whole office when someone found out someone broke something somewhere )
On a more serious note,
well done!
edit: for those who don't know, "blame" is a Subversion feature that tells you for every line in a file who modified it last.
|
Lebaneur
|
Posted - 2009.04.08 16:13:00 -
[14]
Excellent work! Now go back to Fix-what-ever-is-wrong_with-FW!
|
Mors Magne
M. Corp Mostly Harmless
|
Posted - 2009.04.08 16:37:00 -
[15]
Good work!
It's nice to hear the story of what happened. The NEW M.Corp Data Hub - Check it out! |
Zex Maxwell
Caldari
|
Posted - 2009.04.08 17:17:00 -
[16]
NICE! I cant wait to fly into jita to sell my minerals. ---
|
Haradgrim
Tyrell Corp INTERDICTION
|
Posted - 2009.04.08 17:26:00 -
[17]
Cudos --
Originally by: CCP Oveur Just donęt forget the reach-around.
|
The Snowman
Gallente Aliastra
|
Posted - 2009.04.08 18:18:00 -
[18]
So are we back to full operating "as intended" functionality?
Not to diminish the good work, but it still feels like there is a lot more lag, generally, than there was before Apochrypha in and around Jita. Though this maybe unrelated.
|
Solbright
Advanced Security And Asset Protection
|
Posted - 2009.04.08 18:35:00 -
[19]
Originally by: CCP Fallout ... CCP Atlas gives us the lowdown on not only how they searched and destroyed the issue, but how they fixed it... live.
Atlas is lacking datestamps on all that.
|
Slave 2739FKZ
|
Posted - 2009.04.08 19:03:00 -
[20]
CCP, fighting agaisnt their own messy code since 2003. Mad Props, GF!
|
|
Hi Lo
Royal Enterprise
|
Posted - 2009.04.08 20:03:00 -
[21]
YAY, a Dev Blog! Karl Kopalnia > omg Hi Lo is the name I use for all my gaming characters for 15 years--you're the one that took it :*( |
DaiZ Do
|
Posted - 2009.04.08 20:41:00 -
[22]
/me smells geexcitement
well done
|
Tonto Auri
Vhero' Multipurpose Corp
|
Posted - 2009.04.08 20:59:00 -
[23]
One of the few advantages of interpreted language... Well done, guys and gals, well done... -- Thanks CCP for cu |
Glassback
Body Count Inc. Against ALL Authorities
|
Posted - 2009.04.08 22:34:00 -
[24]
Edited by: Glassback on 08/04/2009 22:34:07 nerds.. ;)
|
Serenity Steele
Dynamic Data Distribution Ministry of Information
|
Posted - 2009.04.08 23:09:00 -
[25]
Originally by: Glassback Edited by: Glassback on 08/04/2009 22:34:07 nerds.. ;)
Nah, Nerds don't know how to apply their knowledge.
GEEKS!
≡v≡ Strategic Maps in Eve-Online Store | eve-maps.com |
Backdaft
|
Posted - 2009.04.08 23:28:00 -
[26]
Can anyone from CCP comment further? Specifically, about all the problems that were occurring on the servers the day before the Apoc upgrade downtime?
Jita seemed to be lagging terribly with only 600 players in it that day. Entering and leaving warp the client was having to reset the positions of ships up to 3 times, and this seemed to be happening in many systems. Docking was a long drawn out process as well as if the servers were having trouble syncronizing with the clients.
I saw something that day I've never seen before in EVE - a crash or perhaps severe client disconnect caused a problem where on relogin the client could no longer see the server. I almost never disconnect and I had 2 within the last 3 hours before the Apoc upgrade. After the first one I could only get the clients to see the servers after a computer reboot. Other players in an npc corp chat channel confirmed they had crashed too. The second one occurred for me 10 minutes before the upgrade while I was in Jita trying to reprocess several hundred million isk worth of probes. The result: Client refused to see the servers again before the upgrade started and it turns out the probes never reprocessed themselves.
These probes turned into new probe types worth only 25% of the minerals. I've a petition in on this matter but the GM's are having a hard time confirming that there was any crash or problems with Jita. Without being able to prove a crash or server problem, they can only go by the TOS which I will summurize to state CCP is allowed to change game content at any time and any player losses are just tough luck.
So please, help me out. Can you confirm the servers were "pre-patched" with some Apoc code the day before the official Apoc update downtime? Clearly this would explain why the servers had ships accelerating and deccelerating from warp slower, thus forcing the clients to forcably reposition the ships multiple times. It would also explain the problems with crashes/disconnects and possibly even the lag in Jita that day.
Such confirmation would really help me get my reimbursement petition answered successfully. This has been dragging out for almost a month now and I'd really like to see it closed. But the GM's really need someone at CCP "in-the know" to tell them what was going on that day. Thanks.
|
Backdaft
|
Posted - 2009.04.09 00:18:00 -
[27]
Those one-liners can be a b**ch!
"Dude, I have a date tonight. I've already tested this one line of code, so can you sneak it into your build? I don't have time to check it in myself and then do a full project rebuild before I need to leave."
"You sure you tested it?"
"Yup, and it's just one line. See? Simple."
So Backdaft fatefully adds the one line to his build. He does another complete project update and everything compiles fine. He starts up the game and it seems to run fine. CHECKIN and done. But the next day.
Team Leader: "Management wants me to put this rubber chicken on your door."
Me: "What! I didn't break anything!"
TL: "Well, it turns out this one line of code you added broke the AI and the testers lost most of a night of testing against the newest build."
Me: "Hey, no fair. The chicken is suppose to be for those who checked in code that broke the nightly automated build."
TL: "Management has updated the rules to include breakage of any kind. Don't feel bad. The other programmer admitted it was his fault so we bought a 2nd rubber chicken for him."
Me: "And that's suppose to make me feel better how? But don't worry. I guarantee my next checkin won't be for a long, long time. I'm going to save up all the work from all my completed task and check it in all at once -preferrably only once or twice a month."
TL: "Oh gawd! Please don't do that!"
Me: "Then someone better find a way to get this chicken off my door."
Moral of the story: When someone tells you that Radical Programming and micro-management is the end-all, be-all of coding just remember this: Within 3 months 80% of employees left that game company to find other work. Seems they didn't like 1 hour meetings added on to every work day. They didn't like managers breathing down there necks every 5 minutes for an update. And they didn't like rubber chickens showing up on there office doors for every little mistake.
|
Mara Rinn
|
Posted - 2009.04.09 02:05:00 -
[28]
Edited by: Mara Rinn on 09/04/2009 02:06:02 Real Men take printf debugging out before committing code to production?
Oh, and Backdraft - if it was really "Agile" or "Extreme" programming being applied, the meetings would be 15 minutes of "goals/obstacles" then letting everyone get on with the job.
|
schurem
Silver Snake Enterprise Systematic-Chaos
|
Posted - 2009.04.09 10:28:00 -
[29]
lol you bunch of F'in cowboys!
|
Virgo I'Platonicus
Zoners
|
Posted - 2009.04.09 11:53:00 -
[30]
Lol @ fine blog and nice work :)
V. <3 |
|
Louis deGuerre
Gallente Azure Horizon Federate Militia
|
Posted - 2009.04.09 12:37:00 -
[31]
Damn, too late to post the svn blame joke
|
Grismar
Gallente The Establishment
|
Posted - 2009.04.09 12:42:00 -
[32]
Edited by: Grismar on 09/04/2009 12:43:26 Nice to see a dev blog instead of marketing and hype. I'm continually impressed by the scale of the EVE universe and the way it just keeps going and growing, even though it's serving well over 20x the people at a time it did when I first signed up.
I'm also sure that the "one line of code" that needed fixing was innocent enough by itself, but only caused a problem in the particular context it was living in. This is exactly why questions like "who wrote that line" are generally inconsequential. I understand the euphoria the blog clearly expresses because I know how hard it is to find a line that has nothing wrong with it in itself, but causes a problem in the bigger picture.
Keep up the good work and keep hiring people that get all giddy and talkative when they write good code or fix bad code and I'm sure EVE will endure.
Your EVE IGB home: EVE Wiki, Explorer, Navigator |
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.04.09 13:35:00 -
[33]
"it was running through the CPython interpreter"
That still gives me the shivers. How many people could probably be in Jita if Eve was written in C/C++ or even .Net or Java? 15000? 30000? _________________________ EveAI.Live - The EVE-Online API/class library for .Net, C# and VB.Net |
Rakshasa Taisab
Caldari Sane Industries Inc. United Freemen Alliance
|
Posted - 2009.04.09 14:22:00 -
[34]
Originally by: Amida Ta "it was running through the CPython interpreter"
That still gives me the shivers. How many people could probably be in Jita if Eve was written in C/C++ or even .Net or Java? 15000? 30000?
I have a server written in assembler that can handle 100.000 simultaneous connections. It accepts the connection and then sends keep-alives.
Very fast, very bandwidth efficient.
|
Haniblecter Teg
F.R.E.E. Explorer Wildly Inappropriate.
|
Posted - 2009.04.09 14:53:00 -
[35]
But could that handle 100k people selling, buying, moving 1,000 different item's from a can to hangar, exiting, leaving, changing nodes? ----------------- Friends Forever |
Backdaft
|
Posted - 2009.04.09 17:12:00 -
[36]
Mara, you're right Extreme programming morning meetings are only suppose to be 15 mins -no matter what. But that only works up until you have a manager in the room. Managers can't do anything in 15 minutes. lol I'm thinking there's another joke in there somewhere...something like "And that's what the manager's wife said too!" lol.
|
Backdaft
|
Posted - 2009.04.09 17:26:00 -
[37]
Amida, I'm not familiar with cPython but I do know many higher level languages compile down into C first. Many are highly efficient at this and I suspect any slow downs are minimal -probably less than 5% if I had to guess.
Java is designed to run on all computers by compiling down to a generic bytecode, so until computers have processors that can run the bytecode in hardware, I'd guess Java would be slightly slower. And .Net is all about libraries and reusable code, so if you're not careful you end up with a massive executable, and I suspect some of those libraries can bog you down really bad -so much slower.
In the end, much of it boils down to how efficient and how optimized the language's compiler is. cPython has been around awhile so it's probably pretty good -especially with the guys at ccp pushing that stackless technology forward.
And assembly always wins, although I don't see many people programming in that anymore. The industry moving to O-O, reusable code seems to have killed it. I've not used CUDA, but if assembly is alive anywhere it's probably running on graphics processors, yes? Is CUDA glorified assembly? Or something more thats good enough to supplant assembly even there?
|
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.04.09 20:29:00 -
[38]
Originally by: Backdaft Amida, I'm not familiar with cPython but I do know many higher level languages compile down into C first. Many are highly efficient at this and I suspect any slow downs are minimal -probably less than 5% if I had to guess.
5% would not even be worth to mention. However cPython is WAY slower than that. Compared to C/C++ (e.g. Intel or GCC) cPython can easily be 100fold slower for computationally-heavy tasks. In extreme cases even much more than that. As a rule of thumb in general it will be around 10times slower (that is 1000%).
Originally by: Backdaft
Java is designed to run on all computers by compiling down to a generic bytecode, so until computers have processors that can run the bytecode in hardware, I'd guess Java would be slightly slower. And .Net is all about libraries and reusable code, so if you're not careful you end up with a massive executable, and I suspect some of those libraries can bog you down really bad -so much slower.
Java and .Net are both in the same ballpark. Sometimes they are even faster than C, but as a general rule of thumb you can assume that they are about 30% slower.
Originally by: Backdaft
And assembly always wins
Sure, but the difference between hand-tuned assembly and e.g. C++/.Net isn't that big any more. _________________________ EveAI.Live - The EVE-Online API/class library for .Net, C# and VB.Net |
Sirius Problem
Darkness Inc.
|
Posted - 2009.04.09 20:55:00 -
[39]
Maybe you could deploy one of your teams to the forums. They've been dog slow for weeks now. ---- Train more. Whine less.
|
Franga
NQX Innovations
|
Posted - 2009.04.10 10:23:00 -
[40]
Werd. ----------
|
|
Nomakai Delateriel
Amarr Ammatar Free Corps
|
Posted - 2009.04.11 03:44:00 -
[41]
Bud Light presents: Real men of genius (Real men of genius) Today we salute you Mr.Jita-live-code-debugger (Mr.Jita-live-code-debugger) You've given us the real New Eden dream, a jita with a ping of less than 3 seconds (less than 3000 milliseconds). With your assistance Jita re-sellers can now buy and re-sell goods at 20% higher prices faster than ever before (Zealots for 150 millions). Always making sure that trading takes place in one system, and one system only (Never go outside The Fooooorge!) So, crack open an ice-cold bud light you icelandic code-crusader, you deserve it (Mr.Jita-live-code-debugger). Bud light beer, anheuser-busch, St.Louis Missouri.
P.S: No critique intended. Just a bit of fun. ______________________________________________ -My respect can not be won, only lost. It's given freely and only grudgingly withdrawn. |
Trustworthy Joe
|
Posted - 2009.04.11 17:03:00 -
[42]
Originally by: Nomakai Delateriel Bud Light presents: Real men of genius (Real men of genius) Today we salute you Mr.Jita-live-code-debugger (Mr.Jita-live-code-debugger) You've given us the real New Eden dream, a jita with a ping of less than 3 seconds (less than 3000 milliseconds). With your assistance Jita re-sellers can now buy and re-sell goods at 20% higher prices faster than ever before (Zealots for 150 millions). Always making sure that trading takes place in one system, and one system only (Never go outside The Fooooorge!) So, crack open an ice-cold bud light you icelandic code-crusader, you deserve it (Mr.Jita-live-code-debugger). Bud light beer, anheuser-busch, St.Louis Missouri.
P.S: No critique intended. Just a bit of fun.
i lol'd hard. very nice work sir!
also, GJ CCP _______________________
with a name as trustworthy as mine, i cant POSSIBLY be an alt
Originally by: CCP Navigator I locked this thread BECAUSE OF FALCON
|
Niccolado Starwalker
Shadow Templars
|
Posted - 2009.04.12 09:20:00 -
[43]
Great Dev blog!
Happy Easter to you all!
Originally by: Dianabolic Your tears are absolutely divine, like a fine fine wine, rolling down your cheeks until they flow down the river of LOL |
Mr Goldfish
Ceptacemia
|
Posted - 2009.04.14 08:08:00 -
[44]
Kudos to the CCP team for finding the problem and also posting this type of blog. Being a IT-consultant myself give me great satisfaction to read them as it's these types of things I'm facing every day. Goldie |
Pilk
Blade.
|
Posted - 2009.04.16 12:14:00 -
[45]
Originally by: Amida Ta
Originally by: Backdaft Amida, I'm not familiar with cPython but I do know many higher level languages compile down into C first. Many are highly efficient at this and I suspect any slow downs are minimal -probably less than 5% if I had to guess.
5% would not even be worth to mention. However cPython is WAY slower than that. Compared to C/C++ (e.g. Intel or GCC) cPython can easily be 100fold slower for computationally-heavy tasks. In extreme cases even much more than that. As a rule of thumb in general it will be around 10times slower (that is 1000%).
Originally by: Backdaft
Java is designed to run on all computers by compiling down to a generic bytecode, so until computers have processors that can run the bytecode in hardware, I'd guess Java would be slightly slower. And .Net is all about libraries and reusable code, so if you're not careful you end up with a massive executable, and I suspect some of those libraries can bog you down really bad -so much slower.
Java and .Net are both in the same ballpark. Sometimes they are even faster than C, but as a general rule of thumb you can assume that they are about 30% slower.
Originally by: Backdaft
And assembly always wins
Sure, but the difference between hand-tuned assembly and e.g. C++/.Net isn't that big any more.
You're missing the point, but first I'll address a separate issue--Java compiles to bytecode, yes, but then that bytecodes has been run (since the mid-90's) on a HotSpot-style JVM. That means that, in tight loops, Java's performance exactly equals that of C (which in turn would exactly equal the performance of machine code).
Anyway, it's just as edifying to say, "EVE would run 100,000 times faster on custom silicon instead of generic Intel." While it's doubtless true that some of the tighter code would benefit from being written in a lower-level language, it's an enormous investment of time to retrain dozens of programmers, and not all of them would get it Right (tm). So, let's say you lose 30% of performance by running through CPython. How much do you lose when a novice C programmer implements a terrible sorting algorithm, because unlike in Python, he doesn't have a standard API that includes a good one? How many features do you lose when an expert C programmer spends more time writing bounds-checking code on all his functions than actually providing functionality?
The question isn't whether EVE could be faster if written in a different language, it's whether EVE could: a) exist at all, and b) be better if written in a different language. And given CCP as a backdrop (i.e., not disputing, there are companies out there with the tool- and skillsets already in place that could write an EVE from scratch in C, or SmallTalk, or pure lambda calculus), I think Stackless is the language of EVE going forward.
Now, that said, my understanding is that CCP already looks at tight loops within the game and recodes and recompiles them into PIC that they call from within Python. So they get most of the benefits of a low-level language, at a fraction of the cost.
Honestly, the only thing I would change about CCP's coding environment is to make them understand that an hour of downtime every day is a sign of weakness. Hire four spectacularly-good Stackless programmers and assign them to nothing but cleanup and rewriting duty. Inside of a year, you'll never need to reboot the platform.
--P
Kosh: The avalanche has already started. It is too late for the pebbles to vote. Tyrrax's bet status: PAID! |
Southern Suzy
Minmatar Brutor tribe
|
Posted - 2009.04.20 11:16:00 -
[46]
Originally by: Zex Maxwell NICE! I cant wait to fly into jita to sell my minerals.
Prices arent good there for selling, try remoter area's that will give ya a lot more isk So wait this is the end of my post allready?
I'm not in multiple alliances to spy! I'm in them so I'll always be on the winning team |
Amida Ta
German Mining and Manufacture Corp.
|
Posted - 2009.04.20 22:45:00 -
[47]
Originally by: Pilk
Anyway, it's just as edifying to say, "EVE would run 100,000 times faster on custom silicon instead of generic Intel." While it's doubtless true that some of the tighter code would benefit from being written in a lower-level language, it's an enormous investment of time to retrain dozens of programmers, and not all of them would get it Right (tm). So, let's say you lose 30% of performance by running through CPython. How much do you lose when a novice C programmer implements a terrible sorting algorithm, because unlike in Python, he doesn't have a standard API that includes a good one? How many features do you lose when an expert C programmer spends more time writing bounds-checking code on all his functions than actually providing functionality? --P
The point is that if CPython would be 30% slower nobody would care. But it it more likely 1000% slower, in calculation-heavy tasks (fleet fight maybe?) possibly even 10000%. Surely it's a good thing to choose a point where you get good bang for the buck (which means going straight C is likely not the best solution). Howerver do you really think CCP is 10 or even 100times more effective using Python instead of e.g. .Net? I'm VERY sure they are not.
Sure they have the server already written and because of that I doubt that they WILL NOW restart from scratch and also nobody knows IF they really use pure Python on the server (parts of the client are not Python (for performance reasons)) but that still doesn't make the situation any better. _________________________ EveAI.Live - The EVE-Online API/class library for .Net, C# and VB.Net |
|
|
|
Pages: 1 2 :: [one page] |