Pages: 1 2 3 4 5 6 :: [one page] |
|
Author |
Thread Statistics | Show CCP posts - 19 post(s) |
|
CCP Wrangler
|
Posted - 2008.09.26 17:35:00 -
[1]
For some time now we have been developing StacklessIO to reduce lag in EVE, and it was recently introduced on Tranquility. Explorer has written a blog detailing what this is, how it works and how it decreased lag in EVE. The blog is up for your reading pleasure: StacklessIO or: How We Reduced Lag
Wrangler Community Manager CCP Hf, EVE Online Email |
|
Kerfira
|
Posted - 2008.09.26 17:43:00 -
[2]
Edited by: Kerfira on 26/09/2008 17:47:09 Yay! I love techie blogs
There was quite a few problems last weekend with ghost ships, ships not disappearing after logging out, etc. While there is no direct link to the new network layer, it seems a logical possibility that a major change like this brought a few unintended consequences. You'll probably look into this, I hope...
Originally by: CCP Wrangler EVE isn't designed to just look like a cold, dark and harsh world, it's designed to be a cold, dark and harsh world.
|
LaVista Vista
Conservative Shenanigans Party
|
Posted - 2008.09.26 17:47:00 -
[3]
How well can you scale this?
Otherwise, good stuff.
|
Reptzo
Channel 4 News Team
|
Posted - 2008.09.26 17:48:00 -
[4]
DEV BLOGS YAYAYAYYAYAYAYAY
I'm excited about new techie dev blogs, can you tell?
|
Tipz NexAstrum
Celestial Horizon Corp. Celestial Industrial Alliance
|
Posted - 2008.09.26 17:52:00 -
[5]
Those are some damn fine improvements, good work guys!
|
Cactus Mack
|
Posted - 2008.09.26 17:55:00 -
[6]
DevBlog quote: At 1,400 pilots on Saturday the node hosting Jita ran out of memory and crashed
Yay! With StacklessIO we can p1ss-off nearly twice as many people as before.
|
Haks'he Lirky
Burning Bright Inc.
|
Posted - 2008.09.26 17:58:00 -
[7]
NICE! :)
Fantastic dev blog.
|
|
CCP Explorer
|
Posted - 2008.09.26 18:02:00 -
[8]
Originally by: Kerfira While there is no direct link to the new network layer, it seems a logical possibility that a major change like this brought a few unintended consequences. You'll probably look into this, I hope...
The Core Technology and EVE Software development teams as well as Quality Assurance and Virtual World Operations have all been monitoring Tranquility intensely since we deployed StacklessIO.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Raketefrau
Caldari Deep Core Mining Inc.
|
Posted - 2008.09.26 18:02:00 -
[9]
This all looks good, it'll be nice to see how it affects fleet battles.
The question of RAM remains, however -- if you end up with a 300 vs 300 battle in some backwater 0.0 system, what are the odds that it's going to have enough RAM to handle the load?
In other words, do you plan on upgrading the amount of memory across the board based on your work with Jita?
|
|
CCP Explorer
|
Posted - 2008.09.26 18:09:00 -
[10]
Edited by: CCP Explorer on 26/09/2008 18:09:36
The old network technology limited us to using 32 bit code. We can now move to 64 bit code (we actually did the bulk of that that work this week) and can then add more memory to the cluster and make use of it. The exact plans haven't made yet so I can't tell or promise if or when that will happen. It is a possibility now.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
Gnulpie
Minmatar Miner Tech
|
Posted - 2008.09.26 18:13:00 -
[11]
Just awesome!!
This is really good stuff. Well done and thanks for the good and fast devblog |
Jameroz
Echoes of Space
|
Posted - 2008.09.26 18:13:00 -
[12]
Very intresting blog! StacklessIO ftw \o/
|
Arous Drephius
Perkone
|
Posted - 2008.09.26 18:15:00 -
[13]
Awesome work.
|
WaiKin Beldar
|
Posted - 2008.09.26 18:25:00 -
[14]
Awesome! Good work
|
Miasia
Konstrukteure der Zukunft United Front Alliance
|
Posted - 2008.09.26 18:25:00 -
[15]
Really immpressive how you achived the reduction of memory usage AND network bandwith usage.
I'm really impressed ... *thumbs up*
|
Reptzo
Channel 4 News Team
|
Posted - 2008.09.26 18:26:00 -
[16]
Originally by: CCP Explorer Edited by: CCP Explorer on 26/09/2008 18:09:36
The old network technology limited us to using 32 bit code. We can now move to 64 bit code (we actually did the bulk of that that work this week) and can then add more memory to the cluster and make use of it. The exact plans haven't made yet so I can't tell or promise if or when that will happen. It is a possibility now.
Will there be any kind of performance gains simply by switching to 64 bit code? (other than the obvious more ram) Are you guys switching all coding over to native 64 bit? Is it already switched?
|
Cat Gilligan
Caldari Blair Corporation
|
Posted - 2008.09.26 18:28:00 -
[17]
64 bit code can address a LOT more memory. Also, in theory 64 bit code can execute more instructions per CPU cycle than 32 bit code, thus getting more out of each CPU.
|
Frau Formaggio
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2008.09.26 18:32:00 -
[18]
Awesome.
Got any links to what "StacklessIO" actually is - the techie devblog was a little light on the tech part.
Also - were any performance metrics based on backwater 0.0 large fleet battles gathered?
Oh .. and ..
AWESOME..
|
Alz Shado
Ever Flow
|
Posted - 2008.09.26 18:38:00 -
[19]
Just out of curiosity, what effect (if any) would this have on issues such as desynch? //// ---------=== []= ---------=== \\\\ Rifter(RedBad)
"Kill a man one is a murderer; kill a million, a conqueror; kill them all, a God." -- Jean Rostand |
|
CCP Explorer
|
Posted - 2008.09.26 18:38:00 -
[20]
Originally by: Reptzo Are you guys switching all coding over to native 64 bit? Is it already switched?
We did a lot of work this week on 64 bit code and Tranquility is currently running a few nodes with all native 64 bit code.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
Ashen Wraith
|
Posted - 2008.09.26 18:43:00 -
[21]
should have a armageddon day and get a system for just chaos. pushing most people in you can. Check it for real combat... 400 vs 600?
|
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.26 18:47:00 -
[22]
The results certainly look promising, but I would like to know what StacklessIO actually does. Is it a replacement for the standard IO extensions in Python? Does it use IO to do more things concurrently (circumventing the GIL). Or is a replacement of your own IO implementation within EVE. Or am I totally off the mark?
Inquiring minds would like to know ... -- Quis custodiet ipsos custodes? |
DigitalCommunist
Obsidian Core
|
Posted - 2008.09.26 18:56:00 -
[23]
This blog was good, more like this!
|
Batolemaeus
Caldari Batolemaeus Corp
|
Posted - 2008.09.26 18:57:00 -
[24]
Yay, DevBlog Yay, Tech stuff Yay, pretty figures.
/me hands Explorer a beer
Originally by: CCP Prism X In New Eden, EVE wins you.
|
Raquel Smith
Caldari Freedom-Technologies
|
Posted - 2008.09.26 18:58:00 -
[25]
Way cool!
-- Creator of The Ruby API Library |
Rexthor Hammerfists
Rage of Inferno
|
Posted - 2008.09.26 19:01:00 -
[26]
For someone who has no idea what stackless io is and understands nothing of that blog, how will this effect fleetbattles and about when? -
Boosters and PirateProfessions
|
Chainsaw Plankton
IDLE GUNS IDLE EMPIRE
|
Posted - 2008.09.26 19:03:00 -
[27]
i love you guys and the flood of dev blogs. not sure if it quite makes up for the month of silence though and damn the graphs look sexy, how long till 2000 in jita
|
Trent Nichols
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2008.09.26 19:19:00 -
[28]
Two fantastic Dev blogs in less than a week's time. On top of that there isn't a Gallente nerf in sight.
I just have to say... wtf?
Seriously, it is encouraging to see CCP working at what has been, arguably, Eve's biggest problem.
Logistics deployables mean less grind and more pewpew! |
maltari
Caldari Eve University Ivy League
|
Posted - 2008.09.26 19:21:00 -
[29]
Very interesting blog. As someone working on performance issues on a big website, I feel your pain. Fix one bottleneck, and bam, you just hit another performance issue, and you start all over again... Keep up the good work :)
|
Danjira Ryuujin
Caldari
|
Posted - 2008.09.26 20:00:00 -
[30]
Exciting stuff, Thanks for the update!
Amarr - Annoying the Eve Community since 2005 |
|
Easy Tiger
Caldari
|
Posted - 2008.09.26 20:03:00 -
[31]
OK that is damn impressive
Have to say I've noticed a difference around Jita. Mad props to all involved
|
Idara
Caldari Failure Corp
|
Posted - 2008.09.26 20:04:00 -
[32]
Edited by: Idara on 26/09/2008 20:03:44 That's hot. --- Failure Corp [FAILD] - Failing to fail first
in EVE - Idara |
Solomon XI
Hoist The Colors.
|
Posted - 2008.09.26 20:05:00 -
[33]
I've noticed an improvement lately but have yet to put it to a true test. I <3 Tech dev-blogs, though. ~Solo Hoist The Colors. (CEO) |
Syekuda
Caldari Perkone
|
Posted - 2008.09.26 20:18:00 -
[34]
Questions for the devs here: I've been thinking after I read your wonderful blog. Is the improvement only in the node or cluster where Jita is ? Also, if not which means its everywhere, will the effects (the final stats) be the same in a system which is empty and then all of a sudden, you see 300 ships using the jump gate ?
I was wondering since I don't know much about the network infrastructure that Eve uses, but is the "limitations" the same with an 0.0 system. In case you don't understand or its really not clear lets say the cluster in Jita as 32gig of ram because its always busy. Would there be the same in a 0.0 system or a system thats not normaly busy ?
|
|
CCP Atlas
|
Posted - 2008.09.26 20:53:00 -
[35]
Edited by: CCP Atlas on 26/09/2008 20:54:46
Originally by: Syekuda Questions for the devs here: I've been thinking after I read your wonderful blog. Is the improvement only in the node or cluster where Jita is ? Also, if not which means its everywhere, will the effects (the final stats) be the same in a system which is empty and then all of a sudden, you see 300 ships using the jump gate ?
I was wondering since I don't know much about the network infrastructure that Eve uses, but is the "limitations" the same with an 0.0 system. In case you don't understand or its really not clear lets say the cluster in Jita as 32gig of ram because its always busy. Would there be the same in a 0.0 system or a system thats not normaly busy ?
This is a performance boost to all solar systems in that requests you make will reach the server more quickly. The effect is mostly noticeable where the node is heavily loaded and thus Jita has been used as an example.
This will not allow the node to process your requests more quickly though once it reaches the application layer, but you shouldn't have to wait 5 minutes for a module to activate.
|
|
Jakus Cemendur
Caldari Federal Defence Union
|
Posted - 2008.09.26 20:56:00 -
[36]
Edited by: Jakus Cemendur on 26/09/2008 20:56:26 *Gives all the core team devs a big hug for the sheer awesomeness of the blog*
*and buys beers for those strange people who won't want hugs* ------
|
|
porkbelly
|
Posted - 2008.09.26 20:58:00 -
[37]
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are. What it is! |
|
Komen
Gallente Trinity Nova Trinity Nova Alliance
|
Posted - 2008.09.26 21:42:00 -
[38]
And this is why I continue to play this game. In the space of a week, multiple improvements and changes all directed towards reducing lag - some towards Jita specifically, and then this one which ought to significantly reduce lag everywhere in Eve.
Kudos, dev team.
You've secured another year-long subscription from me. (okay, it's not actually due for renewal 'til February or so, but still).
|
Reptzo
Channel 4 News Team
|
Posted - 2008.09.26 21:43:00 -
[39]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. Crazy Server VooDoo magic.
Fixed it for you.
|
Shadoo
North Eastern Swat Pandemic Legion
|
Posted - 2008.09.26 21:47:00 -
[40]
Wow -- a teched up devblog that isn't watered down with too much marketing talk.
Well done (on both the blog and the significant advancements deployed).
|
|
Jumfat Kohlah
|
Posted - 2008.09.26 21:52:00 -
[41]
Fantastic stuff
Would be intensely interested to see if you have affected or changed the network/transport layers (ISO 7 layer model)... and how did you arrive at this... MOAR detail pls :)
|
Alek Row
Silent Step
|
Posted - 2008.09.26 22:13:00 -
[42]
Impressive. Congratulations!!!
|
Hyren
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2008.09.26 22:15:00 -
[43]
A.W.E.S.O.M.E.
________________________________________________ "It is better to keep your mouth shut and appear stupid, than to open it and remove all doubt." - Mark Twain |
Barwinius
Ars ex Discordia
|
Posted - 2008.09.26 22:35:00 -
[44]
Congratulations on the victory. |
Schneiderr
Asgard Schiffswerften Ev0ke
|
Posted - 2008.09.26 22:44:00 -
[45]
interesting, i like those little tech blogs. keep up the good work!
|
Vaal Erit
Science and Trade Institute
|
Posted - 2008.09.26 22:58:00 -
[46]
Edited by: Vaal Erit on 26/09/2008 23:02:31
Originally by: Dev Blog This new network layer reduces network latency and improves performance in high-volume situations, e.g., in fleet-fights
MOAR MOAR MOAR.
*edit* When you click on dev blogs in the left hand control panel, it seems to think it is Oct and you have to go to archive to find the current blog. *edit* --
http://desusig.crumplecorn.com/sigs.html
|
Taedrin
Gallente 24th Imperial Crusade
|
Posted - 2008.09.26 23:19:00 -
[47]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Words in this post that I did not understand:
"tasklets" "Winsock operations" "IO completion ports"
I look forward to that devblog of yours, hopefully you'll be able to showcase all this new tech you guys have been working on. I know I love tech blogs the most, and I'm sure that there's plenty more who do too.
|
mechtech
Deep Space Productions
|
Posted - 2008.09.26 23:22:00 -
[48]
Yep, I love the tech blogs as well, the eve server is an amazing thing.
GOOD JORB GUYS!
|
MotherMoon
Huang Yinglong
|
Posted - 2008.09.26 23:23:00 -
[49]
CCP FIX THE LAG!
why can we only have 1400 people in one fleet battle!
(with much love wrangler don't kill me but someone had to say it :P)
|
Blyghme
Gallente Strohl Munitions
|
Posted - 2008.09.26 23:27:00 -
[50]
"It was not uncommon that client requests could take up to 1-2 minutes to reach the service layer on the server, and the requests would be delayed seemingly randomly since for two requests in succession then the first one could be delayed for minutes while the second one would get a response almost immediately. From a player's perspective this would manifest itself in lag and strange client behaviour as requests were delayed and completed by the server much out-of-order."
OK, I know that this was referring to Jita, but doesn't it also sound a lot like the symptopms of desync?
|
|
Jordan Musgrat
H A V O C
|
Posted - 2008.09.26 23:31:00 -
[51]
What you say is true, market requests and general response in Jita has been absolutely beautiful the past few days. If you can keep this up, and it applying in all systems, then you have done something that we have not seen in years-- reduce what noobs like us call "lag" in a big and significant way. Thank you CCP, now if you can only keep going in this direction, and take a look at the mechanics that cause server limitations to become apparent in the first place, eve will be running on happily for years yet. -----------
Primary is family values, secondary is 0.0... |
Kuzya Morozov
Gallente L8L8L8
|
Posted - 2008.09.26 23:32:00 -
[52]
Can't really say anything about Jita, only use it a few times a month, and no matter by how much you increase the limit, people will max it out.
However, I am looking forward to see if this effects fleet fights, good work CCP :)
|
|
CCP Explorer
|
Posted - 2008.09.26 23:34:00 -
[53]
We did a lot of work last year in improving the client/server communication on the physics simulation. The old network technology could possibly have caused or contributed to instances of desync. StacklessIO will help in all circumstances.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Kyguard
Forged from Fire
|
Posted - 2008.09.27 00:10:00 -
[54]
Excellent work - very pleased -
|
Blyghme
Gallente Strohl Munitions
|
Posted - 2008.09.27 00:39:00 -
[55]
Thanks for the quick reply.
Excellent work on the new network stack :)
|
Gun Hog
Caldari Ardent Industrial Hydra Alliance
|
Posted - 2008.09.27 02:21:00 -
[56]
All in all, these are pure golden improvements!
Enjoy the reduced lag...while it lasts! In about a month of two, people who avoided Jita are going to start going, and you shall have the same problem again. Local is going to exceed 2K, pilots are going to have more fights outside of 4-4, and market scams will plague the local chat at a whole new level.
The problem, sadly, is not that the node cannot handle the load...it is because people keep increasing the load until lag and crashes happen! I suppose it is the same as the theory that says that if you expand a heavily used highway, you could end up making the congestion worse. _______________________________________________ The Original Ultranoob 0.0 Carebear!!
Originally by: CCP Wrangler
I think I just trolled against my own company though...
|
Todd Overbeck
|
Posted - 2008.09.27 02:49:00 -
[57]
Originally by: Taedrin
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Words in this post that I did not understand:
"tasklets" "Winsock operations" "IO completion ports"
I look forward to that devblog of yours, hopefully you'll be able to showcase all this new tech you guys have been working on. I know I love tech blogs the most, and I'm sure that there's plenty more who do too.
A "tasklet" is a single unit of computation in stackless python.
"Winsock operations" and "IO completion ports" refer to Windows NT TCP/IP programming interface internals. Basically he's talking about a method of having the windows kernel notify the software with a window message when a request is completed, instead of the software having to A) poll periodically to see if it was completed (aka "blocking"), or B) check if it was completed or not before attempting to send another message. (aka "non blocking")
In short, a more efficient way of doing things.
|
Athanasios Anastasiou
The Illuminati. Pandemic Legion
|
Posted - 2008.09.27 02:57:00 -
[58]
Edited by: Athanasios Anastasiou on 27/09/2008 02:58:53 Asynchronous stuff with threads sounds like a pain to get stable, especially for something this complex. GJ CCP.
|
Karbowiak
Caldari coracao ardente Triumvirate.
|
Posted - 2008.09.27 03:21:00 -
[59]
So, this must be why i hardly felt any lag in JU- tonight with ~300 in system doing battle..
epic.. :D
FLEET BATTLES HERE I COME \o/
|
Jacob Goods
|
Posted - 2008.09.27 05:19:00 -
[60]
One of the more interesting articles I've read in very long time. Thanks!
|
|
Beltantis Torrence
|
Posted - 2008.09.27 06:15:00 -
[61]
Impressive stuff guys, well done.
|
Suitonia
Gallente interimo
|
Posted - 2008.09.27 06:29:00 -
[62]
Nice work.
|
Franga
NQX Innovations
|
Posted - 2008.09.27 07:21:00 -
[63]
Originally by: DigitalCommunist This blog was good, more like this!
He speaks truth. This is the blog we were looking for!
GOOD DEV BLOG! Great improvements by the looks of thing. I wait to hear what it's like in the bigger fleet fights of FW and 0.0 alliances.
/me touches CCP. ----------
|
Disteeler
Primera Fundacion
|
Posted - 2008.09.27 07:22:00 -
[64]
wow those blogs makes a difference! nice technology. Couple that with infiniband and you have a winner model of mmo server farm
|
Malcanis
RuffRyders Axiom Empire
|
Posted - 2008.09.27 07:29:00 -
[65]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Also +1 for having a great nick
|
Ancy Denaries
Caldari Amarr Sisterhood of Galactic Sirens
|
Posted - 2008.09.27 07:40:00 -
[66]
Most impressed both by the improvements and the devblog. I approve of this service :D Very very nice to hear that this benefits the whole cluster, and not just a few nodes. Very nice work and congratulations on pulling it off!
Balance is important, but you will always adapt to changing circumstances and you don't whine about stuff you can't change. |
Chienka
Victory Not Vengeance
|
Posted - 2008.09.27 08:45:00 -
[67]
Any more accounts on how much this is affecting the lag in blob warfare?
|
Brock Nelson
Caldari Flux Technologies Inc
|
Posted - 2008.09.27 09:16:00 -
[68]
Edited by: Brock Nelson on 27/09/2008 09:19:41 I'm looking at the Average Cluster Ping Time chart and one question about them.
The spike using old technologies between 20:24 and 21:36, would you say that is due to the number of player during that time frame?
If that's the case, how is it possible that the stacklessio maintains the average ping time regardless of the number of player in Jita? Edit: Would the average ping not correlate to the number of players in Jita?
Also, CCP Explorer mentioned that some of the node are already running 64 bit; how does the performance of those node compare against the 32 bit nodes? I know that performance data may be biased (ie: more load in one node vs another).
I was wondering what exactly is causing performance issue in any node or in this case, Jita. I know its due to 1400-something people but does the demand on the node break down into something more specific? Such as people putting more people on block b/c of scams? People accessing Jita market? Players just sitting in station? Chatroom? etc
Don't forget to copyright "StacklessIO"
|
iudex
Caldari State Protectorate
|
Posted - 2008.09.27 09:17:00 -
[69]
I don't know if this has something to do with the new technology, but i noticed a new problem which started few days or weeks ago.
Although there is no lag in my mission hub (except when reprocessing big ammounts of modules, which now can take up to 5 minutes), the autorepeat modules sometimes (actually quite often last days) have a strange ******ation: the launcher rof can be up to double as much, it sometimes take up to 15 seconds between the launch of a missile, where the theoretical rof (the one shown on launcher info) is 8.3 sec. Also the shield booster take more seconds to activate, when there is no such effect my perma-boosting shieldtank remains cap-stable at 40%, with this new strange lag effect it's sometimes up to 70-80% (which can only mean that the shieldbooster rof is much higher than usual). Next to that i don't see any lag, although there are over 100 people in that mission hub (Irjunen) the fps stays around 40-60.
Maybe this has nothing to do with StacklessIO, but this phenomenon is rather new, it's hard to detect for missionrunners that don't use permatanks and fof launchers (if you reactivate them a lot you might not notice the actual rof increase), so maybe there are connections between that new phenomenon and the new technology.
_________________________________________ Faction Standings: Serpentis +7.60 // Angel Cartel +7.31 // Minmatar Republic -8.56 // Gallente Federation -9.71
|
Brock Nelson
Caldari Flux Technologies Inc
|
Posted - 2008.09.27 09:22:00 -
[70]
inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
|
|
Vim
Spook Division
|
Posted - 2008.09.27 09:56:00 -
[71]
I approve of what your doing with my sparkling purple me!
|
|
CCP Explorer
|
Posted - 2008.09.27 10:31:00 -
[72]
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied everywhere on the cluster. It is also coming to a client near you next Tuesday when EVE Online: Empyrean Age 1.1.1 will be released.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Lingorm
C C P
|
Posted - 2008.09.27 10:31:00 -
[73]
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied to all node in the Tranquility Cluster, not just to high load nodes.
The 64bit compile of EVE has been deployed to some high load systems and we are actively monitoring the performance of these systems.
CCP Lingorm CCP Quality Assurance QA Engineering Team Leader
Originally by: Lord Fitz Eve is to WoW as Wow is to an 8 player game of Unreal Tournament.
|
|
Onyx Asablot
M. Corp Mostly Harmless
|
Posted - 2008.09.27 10:50:00 -
[74]
Great job, thank you very much CCP.
The NEW M.Corp Data Hub - Check it out! |
Radeberger
Caldari I Care...... Seriously i do
|
Posted - 2008.09.27 11:03:00 -
[75]
After reading the dev blog it came to me that it seems you now have a way to measure lag.
If that is the cause will that mean that you will use this on a more permanent basis? Will that also mean that you can begin to reimburse people who lose their ships etc from laggy fleet fights?
Other than that sure looks impressive, very nice work.
|
|
CCP Explorer
|
Posted - 2008.09.27 11:03:00 -
[76]
Edited by: CCP Explorer on 27/09/2008 11:04:12
Originally by: Brock Nelson The spike using old technologies between 20:24 and 21:36, would you say that is due to the number of player during that time frame? If that's the case, how is it possible that the stacklessio maintains the average ping time regardless of the number of player in Jita? Edit: Would the average ping not correlate to the number of players in Jita?
The old technology exhibited very particular behaviour when the number of pilots crossed a certain threshold. StacklessIO provides much better performance that is less correlated to player numbers.
Quote: Also, CCP Explorer mentioned that some of the node are already running 64 bit; how does the performance of those node compare against the 32 bit nodes? I know that performance data may be biased (ie: more load in one node vs another).
It's too early to tell what the performance gain will be, we have had native 64 bit code running for only a few days now on selected nodes in the cluster. The biggest advantage is scalability, the ability to use more memory.
The normal setup in the cluster is that a blade has two 64 bit processors, 4 GB of memory and runs Window Server 2003 x64 (we are planning an upgrade to Windows HPC Server 2008). Each blade runs two nodes and each node then hosts a number of solar systems. There are also dedicated nodes for the market (each market blade runs three market nodes), dedicated nodes for corporation services, a dedicated head node for the cluster, etc. Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and three other solar systems are assigned to that pool. That pool is now running all native 64 bit code and the blades have been upgraded to 16 GB of memory.
Quote: I was wondering what exactly is causing performance issue in any node or in this case, Jita. I know its due to 1400-something people but does the demand on the node break down into something more specific? Such as people putting more people on block b/c of scams? People accessing Jita market? Players just sitting in station? Chatroom? etc
... all of the above? The Inventory System is high on the list in Jita as players move a large number of items between hangar and cargo. The Agression Manager and Damage Tracking and Calculation System are low on the list in Jita.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Mors Magne
Caldari M. Corp Mostly Harmless
|
Posted - 2008.09.27 11:06:00 -
[77]
This is all very exciting!
|
Whiskey Girl
|
Posted - 2008.09.27 11:12:00 -
[78]
nice now for fleet battles that should be a improved area
|
ArchenTheGreat
Caldari D00M. Triumvirate.
|
Posted - 2008.09.27 11:22:00 -
[79]
Originally by: Taedrin
"tasklets" "Winsock operations" "IO completion ports"
Tasklets - as far as I remember they are light threads, something which Stackless python is good for Winsock - http://en.wikipedia.org/wiki/Winsock IO completion ports - http://technet.microsoft.com/en-us/sysinternals/bb963891.aspx
|
Dudley Beekle
|
Posted - 2008.09.27 11:32:00 -
[80]
Nice work CCP, good to see theirs life left in the cluster yet
|
|
Lucas Avignon
Avignon Associates Inc.
|
Posted - 2008.09.27 11:43:00 -
[81]
Best dev blog of all time \o/
Wow this is amazing, I was going to say was the memory problem to do with 32bit code but it seems you are already deploying 64bit code server and client wide.
I had become impatient with CCP about lack of info regarding your tackling of the lag problem, of course now there are blogs coming left right and center and actual solutions being implemented to reduce lag and improve the server performance.
There have been a lot of dev posts on this issue lately which is great and I look forward to hearing more, especially with regards to client/server code changes that can allow multi threading and allow Solar systems to run dynamicaly on multiple cores.
As well the code changes that will allow the implementation of infiniband and what is planned for the server hardware upgrade.
For me this is the most interesting and important project in Eve, far overshadowing anything ambulation will bring.
Originally by: CCP Prism X Yeah, and while we're at it we can create a controlled environment around account hacking and credit card fraud and all the other EULA breaches..
|
DaDutchDude
Minmatar Sebiestor tribe
|
Posted - 2008.09.27 12:00:00 -
[82]
Holy [censored] Batman! Dev Goodness Galore!
1) Very nice blog and nice dev responses with lots of new info! Kudos to CCP!
2) Very nice improvements! It looks like some important problems for not only Jita but also fleet battles have been squashed! Double kudos CCP, esp. to the developers in the basework doing the magic server voodoo stuff nobody is usually aware of.
3) Isn't the name StacklessIO a bit ... uhm ... confusing? As far as I understand is (or guesstimate), you picked the name since it fits in nicely with the features Stackless Python offers, although it's actually more about non-blocking IO by adding more support asynchronous and parallel communications. Am I right?
4) Memory usage is the new bottleneck? Halp! a) I guess for systems like Jita and the other one node systems, it is easy to predict the memory load will be high, so with 64-bit code and 16GB, this seems well under control. But how about massive 0.0 battles? Now that the network IO is no longer the bottleneck, this could become a problem with the unpredictable nature of massive 0.0 fleets. From what I understand, they could still cause out of memory problems in systems that have less hardware available. I know the memory usage has been lowered by your hard work over the past week, but are you taking aditional steps to prevent this? Are more systems being updated with more memory and 64bit code, or do you have another trick up your sleeve? And can you make failures more graceful perhaps then just a node crashing?
b) I'm noticing that the memory usage on the nodes on the last graph only goes up and never back down until downtime, despite the fact that downtime is not the busiest time of the day. It makes me wonder about memory leaks and if this is the reason you require a daily server reboot. Could you tell us more about that?
5) Removing old bottlenecks --> find new ones! I know from personal experience that removing one bottleneck can lead to the next system to fail. We already seen the memory issue, but are there any other things you found out about your systems / software / infrastructure by speeding up IO?
6) Windows Server 2008 HPC: ORLY? I read somewhere about 2008 HPC and it looks like it would fit well with CCP, but can you tell us more about what you are specifically hoping to gain from this upgrade?
Looking forward to your responses!
|
Deva Blackfire
D00M.
|
Posted - 2008.09.27 12:01:00 -
[83]
Originally by: Karbowiak So, this must be why i hardly felt any lag in JU- tonight with ~300 in system doing battle..
epic.. :D
FLEET BATTLES HERE I COME \o/
Pretty much same question - was JU- covered by this "magic stuff" yesterday? We had 350 on local and lag was minimal (at one point it spiked for me and blocked EVERYTHING for like 10seconds but after this highest lag i had was 3-5 seconds). Also we were using drones around - launching drones mid fight took me 15-20 seconds tops.
|
Blazde
4S Corporation Morsus Mihi
|
Posted - 2008.09.27 12:42:00 -
[84]
I'll hold off full enthusiasm until the long term effect on fleet battes is clearer but it's most definetely fantastic to see core technology being ovehauled like this, and cool devblogs to boot
Couple of observations so far:
Of the two big battles I've experienced since the change (WH-JCA on 16th 22:00 - 01:00, local peaked at 440, and M-OEE8 21st 21:00 - 22:00, local around 250) both had very different lag conditions to what I'd usually expect. My ship was generally more responsive and client seemed to stay updated more often. No staring at an empty grid for 5 minutes before loading, no 5 minute activation lag.
However there was plenty of new weirdness, in the first: - Several people jumping in to system and ending up inside the station, ship intact - Several people blackscreening for an hour during login/undock and their ship only appearing ingame when they logged of - One report of a ship remaining in space for 45 minutes (after which GMs intervened)after log-off in a nearby quiet system after leaving the battle. This bug isn't totally new but I've never seen it so extreme as this. - A few reports of 'ghost' ships. A good example:
Quote: the systems around there are still quite messed up. We've been probing and destroying ships that logged off hours ago. When they blow up, a wreck appears, but the ship remains there and becomes unlockable. There's a bob hurricane that has been sitting on a gate in WH for several hours now, and a few jumps down towards TVN, a phantom tri gang on a gate. You jump in and there's about 5 or 6 ships sitting around the gate, all unlockable, and none of them are in local. If you bump them, they slowboat back to where they were.
In the second battle and a very similar one about an hour before there was: - Huge numbers of unlockable 'ghost' ships, mostly where ships had died. - Two occurences of what felt like an almost-but-not-quite node death, requiring everyone to relog during which many dead (and damaged) ships returned to full health, in many cases after insurance had already been paid for the lost ships. - Around 50-75% of killmails never being generated.
I hate to say it but it was categorically worse than before StacklessIO was introduced. I'm assuming it's minor teething problems and will be (or has been) quickly diagnosed and fixed, I'm just saying don't be too fast to relax and sign it off as a job well done
More generally I'm little concerned about your attention to Jita. Lag conditions in jita are presumably pretty repeatable, or predictable rather, so it serves as a good base line metric to measure changes with. But I really wish you'd analyse fleet battles (lowsec FW and 0.0) more closely than it always appears you do. The weird unexpected bugs and lag conditions are more likely to crop up in fleet warfare because players tend to create less common place, 'extreme' node stress conditions (as a simple example it will never happen that hundreds of ships simultaneously activate mwd in close proximity in Jita).
There are still big issues with excess lag created by carriers. There are very common desyncs that occur when capital sized ships collide that put people's supercaps in danger on a daily basis. That ships often remain in space after log off longer than they should in lag is also very serious. We as players can't submit simple repro steps and get these things fixed. It requires careful logging and analyse of real fleet situations server side.
I strongly suspect that more attention to these situations will uncover performance bugs the improve all of EVE. Even if they don't, fleet battles are simply a more critical aspect of EVE than the performance of Jita. At the end of the day I can schedule my trade activities for off-peak times and a little lag in Jita is no big deal compared to it's affect on territorial warfare.
But like I said thumbs up, good to see substantial changes. _
|
Deva Blackfire
D00M.
|
Posted - 2008.09.27 13:27:00 -
[85]
Quote: - Huge numbers of unlockable 'ghost' ships, mostly where ships had died.
O yea that is the "new" problem. Yesterday melee in JU- generated a LOT ghost ships. We had around 30-40 ghost ships on the grid at some point.
|
Felysta Sandorn
coracao ardente Triumvirate.
|
Posted - 2008.09.27 13:31:00 -
[86]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Yeah, I did a similar thing in VB last Tuesday when I was bored at work...
.: A Vagabond's Requiem (Blog) :.
|
Robacz
Essence Trade Essence Enterprises
|
Posted - 2008.09.27 14:06:00 -
[87]
Very nice blog, lots of interesting info! As a Jita resident, I am very happy to hear about this new technology you developed, so please let me give you big fat
GOOD JOB
|
Mallikanth
|
Posted - 2008.09.27 14:21:00 -
[88]
I love techie blogs because of these few points...
1) The 'technical' players who question CCP's revelations about their (CCP's) code and/or servers. They make me laugh.
2) I have not heard of any other Company, let alone a MMO one, who gives as much techie information to the player base as CCP do. Much Kudos to CCP
3) These Techie blogs are always superb. I especially like this one.
Nice one /end fan boy post
|
Disteeler
Primera Fundacion
|
Posted - 2008.09.27 14:44:00 -
[89]
Originally by: CCP Explorer The normal setup in the cluster is that a blade has two 64 bit processors, 4 GB of memory and runs Window Server 2003 x64 (we are planning an upgrade to Windows HPC Server 2008). Each blade runs two nodes and each node then hosts a number of solar systems. There are also dedicated nodes for the market (each market blade runs three market nodes), dedicated nodes for corporation services, a dedicated head node for the cluster, etc. Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and three other solar systems are assigned to that pool. That pool is now running all native 64 bit code and the blades have been upgraded to 16 GB of memory.
Knowledge make customers less whinners, I hope. Anyway, thanks for the info, I love to know tech specs \o/
|
Lor'ath Tylar
Minmatar The All-Seeing Eye G00DFELLAS
|
Posted - 2008.09.27 14:46:00 -
[90]
Originally by: Deva Blackfire
Quote: - Huge numbers of unlockable 'ghost' ships, mostly where ships had died.
O yea that is the "new" problem. Yesterday melee in JU- generated a LOT ghost ships. We had around 30-40 ghost ships on the grid at some point.
I second that. There were heaps of dead ships stuck on the overview. It made harder for the FCs to know which enemy popped and which didn't. :S The lag for me altough was sort of non-existent! The only lag I experienced was when warping to the fight scene and waiting 10-20 secs for the grid to load.
|
|
Nova Fox
Gallente Novafox Shipyards
|
Posted - 2008.09.27 15:05:00 -
[91]
we should ask the outer ring allainces muttually have a massive battle and ask the devs to pay attention to that.
Pre Order your Sisters of Eve ship today!!! |
Popsikle
Minmatar Re-Awakened Technologies Inc Electus Matari
|
Posted - 2008.09.27 15:08:00 -
[92]
Originally by: CCP Lingorm
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied to all node in the Tranquility Cluster, not just to high load nodes.
The 64bit compile of EVE has been deployed to some high load systems and we are actively monitoring the performance of these systems.
When can we get a 64-bit client version? ;p ____ <t20> i want to be in a manager potition at Hooters <SaraDawn> Garthagk, do you have it up ? <Garthagk> I can get it up anytime. |
Smagd
Encina Technologies
|
Posted - 2008.09.27 15:23:00 -
[93]
Awesome work! This wasn't cheap, and it paid out... I love it when that happens.
Question, since a few fellow pilots mentioned Infiniband:
Does this actually go towards Infiniband or will you have to restart from scratch, well with a much better idea on where all the nuts and bolts are?
I fear something the whole network layer will be obsolete with Infiniband, but then I have no real clue how Infiniband works...
|
DaDutchDude
Minmatar Sebiestor tribe
|
Posted - 2008.09.27 15:39:00 -
[94]
Edited by: DaDutchDude on 27/09/2008 15:43:05
Originally by: Popsikle
When can we get a 64-bit client version? ;p
People think 64 bit code magically makes things faster, but 9 out of 10 cases (or perhaps more like 99 out of 100) it doesn't.
Using 64 bit code has one major advantage, which is being able to address more memory. This is especially useful for server applications, since they are often quite heavy on memory usage. Being limited to 4 GB of addressable memory when you're server many users is bad, so scaling up to 64 bit server code is great when you're server as many users as the EVE servers are.
Using 64 bit code for client applications is hardly ever necessary (at least at this point in time), since hardly any single application uses more then 4 GB of memory. I for one would like the EVE client to fit into that category. ;)
One thing people don't realize is there could actually be a downside to using 64 bit code. Dependent on how strictly declare variables in your code, how your compiler handles these variables and how your OS handles 64 bit code, 64 bit code could actually use up quite a bit more memory while not having any performance advantage (using 64 bits instead of 32 bits for the same variable in memory, doubling its memory footprint), possibly even leading to a degradation in performance on computers that are low on memory.
There is one area where 64 bit CPUs and code can lead to performance enhancements, and this is due to parallel processing of numbers in one ALU (Arithmatic and Logic Unit, the actual calculating bit). If you run a program in 64 bit mode using 64 bit code, you can make the ALU do 1 operation on a 64 bit value in one clock tick. However, it is also possible to do the exact same operation (adding, subtracting, etc) on 2 32 bit values at the same time, or 4 16 bit values. This was all introduced by Intel way back under the name MMX, although the CPU was 32 bit so it could only do 2 16 bit values at the same time.
Using this performance enhancements is very hard to accomplish though, and this is why MMX or similar technologies haven't made CPU's twice at fast in 99 out of 100 programs. Your compiler needs to be able to optimize the code to use special instruction sets, and this only has an advantage when you do a lot of the same sort of processing without having to reload different instructions. With stuff like file compression or movie encoding, this technology actually works quite well, but in most programs, there are too many pieces of conditional code (if statements, while statements,for loops, etc), too many dissimilar operations (subtract and add at the same time doesn't work), too many sequence dependencies (first to operation A and then B on one value instead of being able to do them at the same time) and too many different sized values (8 bit booleans, 16 bit short integers and 32 bit floats, etc) for the compiler to optimize for this. And that's even provided the compiler can actually use the MMX type extensions of all CPU's out there.
Personally I think a 64 bit client would for these reasons be useless, and I'd rather have the devs spend time on stuff that actually gets some results while perhaps being less marketable as saying "We have a fully 64 bit client" (which sounds fancy, but actually means nothing in most cases).
PS: Did I explain that right? And yes, I saw the ;P but I thought I'd explain it anyway for people who really think it would be great.
|
Vyktor Abyss
IONSTAR Curatores Veritatis Alliance
|
Posted - 2008.09.27 15:56:00 -
[95]
Great work, backed up by nice stats.
Great too having regular blogs again to read.
One thing fighting the lag monster though - Will you ever really be able to remove all the bottlenecks and "Fix" lag?
A chain is only as strong as the weakest link. And yes, Anne Robinson is a MILF.
|
Blazde
4S Corporation Morsus Mihi
|
Posted - 2008.09.27 16:00:00 -
[96]
By way of an update, as I suspected the stability teething problems aren't solved. We've got a 200 vs 300 battle in MSHD-4 atm and about 500 people staring at the Entering Game screen. 500 would usually be 'playable' but hugely laggy. Whatever improvements in lag we've had have been exchanged for serious stability issues. _
|
|
CCP Explorer
|
Posted - 2008.09.27 16:19:00 -
[97]
Originally by: Popsikle When can we get a 64-bit client version?
Currently this is the server only. On the server there is only piece of 3rd party middle-ware for which we had to acquire a 64 bit version. On the client there are many more middle-ware components. The old network technology blocked 64 bit development so with StacklessIO there is one less hurdle for the client.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Levitikon
Destructive Influence Band of Brothers
|
Posted - 2008.09.27 16:41:00 -
[98]
Edited by: Levitikon on 27/09/2008 16:42:04
Originally by: CCP Explorer
Originally by: Popsikle When can we get a 64-bit client version?
Currently this is the server only. On the server there is only piece of 3rd party middle-ware for which we had to acquire a 64 bit version. On the client there are many more middle-ware components. The old network technology blocked 64 bit development so with StacklessIO there is one less hurdle for the client.
If you have time to answer silly questions on the forums, could you please be so kind and reassign a bit faster node (hint; Jita; hint) to MSHD solarsystem? We already had three node crashes in last 30 minutes, fourth won't make much difference.
|
|
CCP Explorer
|
Posted - 2008.09.27 17:44:00 -
[99]
Edited by: CCP Explorer on 27/09/2008 17:48:01
Originally by: Levitikon If you have time to answer silly questions on the forums, could you please be so kind and reassign a bit faster node (hint; Jita; hint) to MSHD solarsystem? We already had three node crashes in last 30 minutes, fourth won't make much difference.
The system administrators have been monitoring this fight closely. MSHD-4 has now been remapped to one of the dedicated machines (native 64 bit code and Jita-style hardware).
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Pliauga
Gallente Federal Defence Union
|
Posted - 2008.09.27 18:20:00 -
[100]
Oh this is GOOOOOD More devblogs like this please My specific request would be for Client and server side 64bit system development progress, with as many details as possible.
And uh... well... LONG LIVE StaclessIO
---------- DRONE love rulez!! 'mkay?! LONG range/"OUT OF SYSTEM" artillery |
|
Faffywaffy
|
Posted - 2008.09.27 19:08:00 -
[101]
Quote: From a player's perspective this would manifest itself in lag and strange client behaviour as requests were delayed and completed by the server much out-of-order.
Whaa? If your design even allows client requests to be processed out of order, you are doing something very very wrong. There is an infinite number of ways you can screwed up if your requests are being processed out of order.
Seriously guys, I'm not pretending to know everything (or much) about your system, but processing requests out of order in an environment where the order is so important is big nono. I would venture a guess that there are many instances where this caused bad things to happen.
|
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.27 20:15:00 -
[102]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Thank you, this will do ... still would love to see a technical blog about it ofcourse, but I'm glad you're getting such positive results. -- Quis custodiet ipsos custodes? |
Popsikle
Minmatar Re-Awakened Technologies Inc Electus Matari
|
Posted - 2008.09.27 20:31:00 -
[103]
Originally by: DaDutchDude Edited by: DaDutchDude on 27/09/2008 15:43:05
Originally by: Popsikle
When can we get a 64-bit client version? ;p
People think 64 bit code magically makes things faster, but 9 out of 10 cases (or perhaps more like 99 out of 100) it doesn't.
...
Personally I think a 64 bit client would for these reasons be useless, and I'd rather have the devs spend time on stuff that actually gets some results while perhaps being less marketable as saying "We have a fully 64 bit client" (which sounds fancy, but actually means nothing in most cases).
PS: Did I explain that right? And yes, I saw the ;P but I thought I'd explain it anyway for people who really think it would be great.
I know the differences and such, and most of the time there is no performance gain (sometimes its even a drop!) in 64-bit non server apps BUT the only reason I would want a 64-bit client is so I can set the cache option to SUPER HIGH and use 4GB for cache during fleet battles ;) ____ <t20> i want to be in a manager potition at Hooters <SaraDawn> Garthagk, do you have it up ? <Garthagk> I can get it up anytime. |
Chainsaw Plankton
IDLE GUNS IDLE EMPIRE
|
Posted - 2008.09.27 21:14:00 -
[104]
woo! again
However I have some disappointment
I did a bunch of jumps (irjunen to ihakana, on shortest) and did the 1,000,000km emergency warp on just about every gate I jumped through. every solar system load I found myself in warp, or even almost out of warp by the time I loaded the system and my ship.
|
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.27 21:54:00 -
[105]
Originally by: Faffywaffy
Quote: From a player's perspective this would manifest itself in lag and strange client behaviour as requests were delayed and completed by the server much out-of-order.
Whaa? If your design even allows client requests to be processed out of order, you are doing something very very wrong. There is an infinite number of ways you can screwed up if your requests are being processed out of order.
Seriously guys, I'm not pretending to know everything (or much) about your system, but processing requests out of order in an environment where the order is so important is big nono. I would venture a guess that there are many instances where this caused bad things to happen.
Nice barking, wrong tree. Clearly, handling IO sequentially is not an option. IO requests were probably handled (semi-)concurrently in random order using standard IO completion port scheduling. They are used in order by the Python part by blocking threads. Problems occur when there are many IO requests at the same time. Then schedulers can get overloaded, and this is probably what happened. Anyway, there's a better scheduler now, and probably more concurrency, and IO request are handled more efficiently, so these things shouldn't happen anymore.
And there was much rejoicing ... -- Quis custodiet ipsos custodes? |
MotherMoon
Huang Yinglong
|
Posted - 2008.09.27 22:16:00 -
[106]
Edited by: MotherMoon on 27/09/2008 22:20:57
Originally by: CCP Explorer
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied everywhere on the cluster. It is also coming to a client near you next Tuesday when EVE Online: Empyrean Age 1.1.1 will be released.
wait so it's not just a layer on your side but also on our side?
so right now it's only on the jita side but the players in jita aren't using it?
like... wait... ow
Quote: Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and four other solar systems are assigned to that pool. That pool is now running all native 64 bit code and the blades have been upgraded to 16 GB of memory.
just for tech junkies like myself could you let us know if thise upgrade to 16 GB of memory was before or after jita ran out of memory?
if it was upgraded after then this is very good news, and I think you should test it out to see how many players it takes to cap out jita again!
|
Faffywaffy
|
Posted - 2008.09.27 22:30:00 -
[107]
Edited by: Faffywaffy on 27/09/2008 22:33:28
Originally by: Bartholomeus Crane
Nice barking, wrong tree. Clearly, handling IO sequentially is not an option. IO requests were probably handled (semi-)concurrently in random order using standard IO completion port scheduling. They are used in order by the Python part by blocking threads. Problems occur when there are many IO requests at the same time. Then schedulers can get overloaded, and this is probably what happened.
And there was much rejoicing ...
I am not talking about I/O, I'm talking about the processing of the requests themselves (after they've been read by the I/O handling layer).
Obviously the I/O should be asynchronous for performance reasons (which, by the way, is not the same thing as non-sequential, as you seem to be implying). Every request should have a serial number attached to it, which the client increases for every request. The server should then handle these requests in-order. Of course, the serial number is redundant if the connection is over TCP, not UDP (I've no idea which CCP uses).
Allowing requests to be processed out-of-order can cause bad, bad things. Here's a relatively benign example: when tackling a target that is about to warp off, it is necessary to first warp scramble and only then web him (if you web first, the target's max speed is reduced and he warps off immediately, as he is already at 3/4 of the reduced max-speed). If requests are processed out of order, and the two requests are sent close to each other (and they almost always are), the server might end up processing the web request first. I'm sure I can come up with much more dramatic examples, where you lose all your money if market orders are processed out-of-order.
Originally by: Bartholomeus Crane
Anyway, there's a better scheduler now, and probably more concurrency, and IO request are handled more efficiently, so these things shouldn't happen anymore.
There is no such thing as "shouldn't" in this case. From the blurb in the dev blog posting, I understand their design allows requests to be processed out of order. Hoping that you process requests efficiently enough for out-of-order handling to never occur is just irresponsible.
Considering the amount of "WTF? This should not be happening!" moments I've experienced while playing EvE, I think a great chunk of them can be attributed to this issue.
|
|
CCP Explorer
|
Posted - 2008.09.27 22:43:00 -
[108]
Originally by: Bartholomeus Crane Nice barking, wrong tree. Clearly, handling IO sequentially is not an option. IO requests were probably handled (semi-)concurrently in random order using standard IO completion port scheduling. They are used in order by the Python part by blocking threads. Problems occur when there are many IO requests at the same time. Then schedulers can get overloaded, and this is probably what happened. Anyway, there's a better scheduler now, and probably more concurrency, and IO request are handled more efficiently, so these things shouldn't happen anymore.
Exactly, the game logic was still handled in the correct order but client network request were sometimes handled very out-of-order by the server.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.27 22:49:00 -
[109]
Originally by: MotherMoon
Originally by: CCP Explorer StacklessIO was applied everywhere on the cluster. It is also coming to a client near you next Tuesday when EVE Online: Empyrean Age 1.1.1 will be released.
wait so it's not just a layer on your side but also on our side? so right now it's only on the jita side but the players in jita aren't using it? like... wait... ow
The server is now using the new network technology internally but the protocol is backward compatible so it can communicate with the clients that are still using the old technology until Tuesday.
Quote: just for tech junkies like myself could you let us know if thise upgrade to 16 GB of memory was before or after jita ran out of memory? if it was upgraded after then this is very good news, and I think you should test it out to see how many players it takes to cap out jita again!
It was upgraded after.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
MotherMoon
Huang Yinglong
|
Posted - 2008.09.27 23:26:00 -
[110]
Originally by: CCP Explorer It was upgraded after.
oh my :)
|
|
Levitikon
Destructive Influence Band of Brothers
|
Posted - 2008.09.27 23:43:00 -
[111]
Edited by: Levitikon on 27/09/2008 23:43:21 Can't say anything more than - Thanks.
|
PC5
Pink Bunnies C0VEN
|
Posted - 2008.09.28 00:03:00 -
[112]
Edited by: PC5 on 28/09/2008 00:03:46 Finaly a GOOD devblog! Thanks! Please more stuff like this. So now goons spaming local for lag wont help them much ;)))
Are drones having any significant factor in fleet fights after that change? I think they still are making a lot of lag, am i right? Especialy in fleet fights. After that change it would be nice if you look at aspects connected to fleet fights - like POS lag, drones lag, module cycles lag. Maybe there you can improve much? Any solutions/ideas for those aspects?
|
Garok Nor
Blueprint Haus
|
Posted - 2008.09.28 00:23:00 -
[113]
Lemme see if I understand this right... Eve was slow so you CREATED a completely new and original network protocol to deal with the problem???????
That's it... next time somebody tells me I am wasting time with videogames, I'm gonna tell them that I'm actually involved in high end testing of new internet technologies that are propelling us towards a brighter technological future (and I also understand I am providing study material for Economists).
TBQH I'm impressed, and quite excited to be even a tiny part of of an MMO which is pioneering new methods of digital communication. What other MMO's are changing paradigms like CCP is?
Yay CCP keep up the good work. ------------------------------------------------- Items posted by me are in no way a reflection of the policies and/or opinions of my corporation or alliance. {though they maybe really ought to be} |
Faffywaffy
|
Posted - 2008.09.28 00:47:00 -
[114]
Originally by: Garok Nor Lemme see if I understand this right... Eve was slow so you CREATED a completely new and original network protocol to deal with the problem???????
That's it... next time somebody tells me I am wasting time with videogames, I'm gonna tell them that I'm actually involved in high end testing of new internet technologies that are propelling us towards a brighter technological future (and I also understand I am providing study material for Economists).
TBQH I'm impressed, and quite excited to be even a tiny part of of an MMO which is pioneering new methods of digital communication. What other MMO's are changing paradigms like CCP is?
Yay CCP keep up the good work.
I'm sure they did some good work there, but they did nothing of the kind you're describing. They simply rewrote the network I/O code using a more efficient (but harder to implement) design. The protocol is the same.
|
Garok Nor
Blueprint Haus
|
Posted - 2008.09.28 01:19:00 -
[115]
Originally by: Faffywaffy
Originally by: Garok Nor Lemme see if I understand this right... Eve was slow so you CREATED a completely new and original network protocol to deal with the problem???????
That's it... next time somebody tells me I am wasting time with videogames, I'm gonna tell them that I'm actually involved in high end testing of new internet technologies that are propelling us towards a brighter technological future (and I also understand I am providing study material for Economists).
TBQH I'm impressed, and quite excited to be even a tiny part of of an MMO which is pioneering new methods of digital communication. What other MMO's are changing paradigms like CCP is?
Yay CCP keep up the good work.
I'm sure they did some good work there, but they did nothing of the kind you're describing. They simply rewrote the network I/O code using a more efficient (but harder to implement) design. The protocol is the same.
Ya I kinda gathered that as I read through the thread and CCP further explained... but my first impression aside, this is still an amzing feat (especially if it works).
Does this mean that SoonÖ could be this tuesday for many of us? ------------------------------------------------- Items posted by me are in no way a reflection of the policies and/or opinions of my corporation or alliance. {though they maybe really ought to be} |
Tarron Sarek
Gallente Biotronics Inc. Alternative Realities
|
Posted - 2008.09.28 01:49:00 -
[116]
Great stuff.
___________________________________
Balance is power, guard hide it well
"Ceterum censeo Polycarbonem esse delendam" |
Herschel Yamamoto
Bloodmoney Incorporated
|
Posted - 2008.09.28 03:08:00 -
[117]
1400 in Jita(with more to come this weekend, from the sounds of that memory upgrade), essentially lag-free fleet battles, and this is before Infiniband comes online. What's next, cats and dogs living together in harmony?
Seriously, you guys rock. Keep it up. ------------------ Fix the forums! |
Chainsaw Plankton
IDLE GUNS IDLE EMPIRE
|
Posted - 2008.09.28 04:39:00 -
[118]
Quote: Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and four other solar systems are assigned to that pool. That pool is now running all native 64 bit code and the blades have been upgraded to 16 GB of memory.
heh what are the 4 other systems
amarr, motsu, rens, orsulate (or whatever that smelly frenchie system is called)?
|
Aidin Amado
Gulliver Corp Prismatic Refraction
|
Posted - 2008.09.28 07:00:00 -
[119]
Edited by: Aidin Amado on 28/09/2008 07:02:01 Tbh I'm fairly impressed with the fact that Eve actually works at all. I mean, we're 30-40k users logged in on a single shard, and even though you're zoning out the players, the single shard system forces you to keep a tag on everyone, and sometimes everyone congregate in a certain place - like Jita. WoW of course staggers away from this problem by sharding. I doubt there is ever more than a couple of thousand players online in each shard in WoW at any given time, and since they are zoned out into the different lands, you get a fairly smooth experience - most of the time.
In my younger days I sniffed at an MMO, but staggered away in horror when I saw what was involved. I think most people compare MMO's to single player games, and probably even think that physics and game logic and such are handled by the clients. But of course, the clients are nothing more than a window that shows a scene. Almost all of the game handling: io, ai, network, physics, collissions, and so on are handled by the server. This is why we warp right through planets. If the server handled that kind of collissions, we'd all have lag all the time.
And when you get 30-40k users, heck even 1-2k users, you have to find a solution of epic complexity. I mean, how do you handle a simple thing like dropping a can in space considering that whoever is near it must first of all see it, and then be able to approach it and act on it, and this at the same time that it was dropped?
Thus I'm quite impressed with anyone that actually manages to make a working one - even though it glitches at time. So good work, guys and girls.
But fix the lag! :P ------ Recruitment Director, Gulliver Corporation |
Feng Schui
Minmatar Ghost Festival
|
Posted - 2008.09.28 07:37:00 -
[120]
Originally by: Aidin Amado Edited by: Aidin Amado on 28/09/2008 07:02:01 Tbh I'm fairly impressed with the fact that Eve actually works at all. I mean, we're 30-40k users logged in on a single shard, and even though you're zoning out the players, the single shard system forces you to keep a tag on everyone, and sometimes everyone congregate in a certain place - like Jita. WoW of course staggers away from this problem by sharding. I doubt there is ever more than a couple of thousand players online in each shard in WoW at any given time, and since they are zoned out into the different lands, you get a fairly smooth experience - most of the time.
In my younger days I sniffed at an MMO, but staggered away in horror when I saw what was involved. I think most people compare MMO's to single player games, and probably even think that physics and game logic and such are handled by the clients. But of course, the clients are nothing more than a window that shows a scene. Almost all of the game handling: io, ai, network, physics, collissions, and so on are handled by the server. This is why we warp right through planets. If the server handled that kind of collissions, we'd all have lag all the time.
And when you get 30-40k users, heck even 1-2k users, you have to find a solution of epic complexity. I mean, how do you handle a simple thing like dropping a can in space considering that whoever is near it must first of all see it, and then be able to approach it and act on it, and this at the same time that it was dropped?
Thus I'm quite impressed with anyone that actually manages to make a working one - even though it glitches at time. So good work, guys and girls.
But fix the lag! :P
this
Project:Gank
Pilgrim Guide
|
|
|
CCP Explorer
|
Posted - 2008.09.28 10:46:00 -
[121]
Originally by: PC5 Are drones having any significant factor in fleet fights after that change? I think they still are making a lot of lag, am i right? Especialy in fleet fights. After that change it would be nice if you look at aspects connected to fleet fights - like POS lag, drones lag, module cycles lag. Maybe there you can improve much? Any solutions/ideas for those aspects?
Currently there is a team of developers in Need for Speed sprints measuring and looking at ways to optimise the code. One of the things they are investigating is drones/fighter vs. turrets/launchers, since it's our impression that drones/fighters contribute less to CPU cycles than turrets/launchers and we want to optimise that code.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.28 10:51:00 -
[122]
Originally by: Chainsaw Plankton
Quote: Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and four other solar systems are assigned to that pool.
heh what are the 4 other systems amarr, motsu, rens, orsulate (or whatever that smelly frenchie system is called)?
The only market hub on a dedicated machine is Jita, the other systems are missions-running systems such as Motsu and Saila.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Thraxon
Black-Sun Frontal Impact
|
Posted - 2008.09.28 13:11:00 -
[123]
This is very good news. Will be looking forward how this will help in fleet battles.
|
Herschel Yamamoto
Bloodmoney Incorporated
|
Posted - 2008.09.28 14:37:00 -
[124]
Originally by: CCP Explorer The only market hub on a dedicated machine is Jita, the other systems are missions-running systems such as Motsu and Saila.
Do you keep one held back for fleet battles, or other short-duration, high-lag events of that type? Or do you pull Saila's machine in favour of AB-CDE's battle for the couple of hours it's needed for? ------------------ Fix the forums! |
Zex Maxwell
Caldari
|
Posted - 2008.09.28 15:13:00 -
[125]
Originally by: CCP Explorer
Originally by: Reptzo Are you guys switching all coding over to native 64 bit? Is it already switched?
We did a lot of work this week on 64 bit code and Tranquility is currently running a few nodes with all native 64 bit code.
WOOT!
64BIT FTW! ^^
|
Callistus
Reikoku Band of Brothers
|
Posted - 2008.09.28 15:16:00 -
[126]
Edited by: Callistus on 28/09/2008 15:16:20
Originally by: Herschel Yamamoto
Originally by: CCP Explorer The only market hub on a dedicated machine is Jita, the other systems are missions-running systems such as Motsu and Saila.
Do you keep one held back for fleet battles, or other short-duration, high-lag events of that type? Or do you pull Saila's machine in favour of AB-CDE's battle for the couple of hours it's needed for?
As I understand it CCP can't reassign systems to different nodes while they're actually online. Load balancing usually only occurs during downtime based on previous activity in the system, so I imagine its very rare for a 0.0 system to be put on a dedicated machine.
Last nights antics in M-O would be the exception; they were able to assign the system to one of the new beefy machines because the node was repeatedly dying anyway. ------------
|
Zex Maxwell
Caldari
|
Posted - 2008.09.28 15:24:00 -
[127]
Originally by: CCP Explorer all of the above? The Inventory System is high on the list in Jita as players move a large number of items between hangar and cargo. The Agression Manager and Damage Tracking and Calculation System are low on the list in Jita.
PvPs be warned.
|
Xavier Zyrae
Demonic Retribution G00DFELLAS
|
Posted - 2008.09.28 17:37:00 -
[128]
Awesome job CCP! I was in doing some shopping in Jita earlier this weekend, before I had seen the blog, and I noticed right away it felt much more 'snappy' with >800ppl in local than it ever did before.
I haven't been in any fleet battles lately but if the improvements in Jita are anything to go by I'm really looking forward to, we've been wishing for this for years... _________________________________________________________
|
london
Jericho Fraction The Star Fraction
|
Posted - 2008.09.28 18:41:00 -
[129]
You guys are sure horny for bar charts. I dig it too, but it's kind of funny when you think about what gets us excited.
|
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.28 19:42:00 -
[130]
Originally by: CCP Explorer
Originally by: Bartholomeus Crane Nice barking, wrong tree. Clearly, handling IO sequentially is not an option. IO requests were probably handled (semi-)concurrently in random order using standard IO completion port scheduling. They are used in order by the Python part by blocking threads. Problems occur when there are many IO requests at the same time. Then schedulers can get overloaded, and this is probably what happened. Anyway, there's a better scheduler now, and probably more concurrency, and IO request are handled more efficiently, so these things shouldn't happen anymore.
Exactly, the game logic was still handled in the correct order but client network request were sometimes handled very out-of-order by the server.
Thanks, I wasn't entirely sure I was correct, hence all the 'probably's.
To 'faffywaffy', it appeared to me that you were advocating synchronous, or blocking IO. Obviously, this is not an option, as it would slow down EVE to a crawl. Asynchronous IO implies some concurrency, as a thread (or tasklet) gets blocked waiting for the IO to finish, but another thread (or tasklet) would be permitted to continue. It is common in Python (and other languages) to handle this through 'extensions'. The important bit here is that all threads for a single timeslice (or equivalent) would have to finish before a new timeslice is started. As such, the game logic would be handled in sequence. Hence the problem when one of those IO requests would take too long to complete. In short, IO requests not-in-order, game-logic in-order.
That was not the problem here though. What I gather from the blog and this thread is that with the old system, in really busy systems like Jita, so many IO requests would hit the network layer that it could not cope (basically thrashing the system). As a result IO requests would be handled way out-of-order and take too long to finish, thereby lagging out the whole system from the network layer up. The new network layer appears to be more scalable and more efficient overall, being able to handle more players in systems, and overall, reducing lag. There's probably an upper limit to this as well, but if it is high enough, who cares!
How this was achieved, I don't know, but I can venture some guesses (mostly based on Porkbelly's reply): more scalable and efficient IO extension(s) written for Python (allowing it to be used as regular blocking operations for tasklets (threads) in stackless Python, meaning it's much easier to use by programmers), better scheduler for IO completion port's messaging queue, and more efficient and lightweight WinSock implementation (probably specifically written for the task). The increase in performance is probably a result of the combination of these things. I'm not quite sure how 64bit memory fits into this, but I can see how more available memory would help.
If you're into these things, it's actually quite exciting, so I still would love to see a technical blog on the actual changes made to the network layer (substituting my wild guesses), but I guess the guys at the Core Server Team will be busy taking more measurements on how the framework actually performs 'in the wild' before they'll make any hard claims.
One thing I would like to point out though: from experience, these types of problems are notoriously hard to pin-point within a complex system, as they are usually transient and usually depend on specific circumstances. In this case, maybe we should be grateful to have a hellhole like Jita. I'll go wash my mouth now ... -- Quis custodiet ipsos custodes? |
|
Faffywaffy
|
Posted - 2008.09.28 21:30:00 -
[131]
Originally by: Bartholomeus Crane To 'faffywaffy', it appeared to me that you were advocating synchronous, or blocking IO.
Originally by: Faffywaffy Obviously the I/O should be asynchronous for performance reasons
Obviously, you did not bother to read my post. Once more - I am not talking about the network I/O layer. I am talking about the request processing (game logic handling) layer. The game logic layer must not, by design, be able to process requests out-of-order.
Originally by: Bartholomeus Crane In short, IO requests not-in-order, game-logic in-order.
That is the desired effect, but the blurb on the dev blog seems to imply that the game logic handling layer can process requests out of order. If it were only the I/O layer that could read requests out-of-order, it could not possibly "manifest itself in strange client behaviour as requests were delayed and completed by the server much out-of-order."
|
Lewis Atreides
Caldari House Atreides trade
|
Posted - 2008.09.28 22:19:00 -
[132]
Easy sulution to population growth problems:diffrent servers.
CCP's sulution:a 2years of work to keep everyone on one sever.
I do kind of wonder if anyone at CCP HQ made a video of the devs actions during the Jita crisis.I know that asking about such a video is a bit nuts but it would be interesting to see how CCP's teames act during a unexpected development.
**--------**--------------** While blood, carnage, and distruction may be my "happy place" that is nothing like the feeling I get while hauling 2 mill isk in goods thru low sec. |
The Jugganaut
|
Posted - 2008.09.28 22:22:00 -
[133]
Originally by: CCP Explorer Currently there is a team of developers in Need for Speed sprints measuring and looking at ways to optimise the code. One of the things they are investigating is drones/fighter vs. turrets/launchers, since it's our impression that drones/fighters contribute less to CPU cycles than turrets/launchers and we want to optimise that code.
I noticed you saying Sprints there, does that mean you work with the SCRUM development methodology? Also.. Do you use any special programming/coding specific approaches such as XP and test first approaches? Basically any of that shit my teachers have been knocking into my skull over the last couple of years?
Must say blogs and forum replies like those from the DEVS here, is why I love the CCP devs!! Goodluck getting blizzard to write stuff like this! :D
|
|
CCP Explorer
|
Posted - 2008.09.28 23:40:00 -
[134]
Edited by: CCP Explorer on 28/09/2008 23:40:44
Originally by: The Jugganaut I noticed you saying Sprints there, does that mean you work with the SCRUM development methodology?
We use Scrum for development of new features (especially where we are venturing into the unknown) and for optimisation tasks where cross-functional teams from different departments are needed. We use Staged Deliveries for more known development tasks and we when have a lot of small features (a category called "General Improvements" in the expansion plan). For defects we allocate time each week and then timebox defect work at regular intervals. The EVE Software Group is also responsible for live software maintenance of Tranquility and all live issues trump any work we are doing.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
DaDutchDude
Minmatar Sebiestor tribe
|
Posted - 2008.09.29 01:05:00 -
[135]
Originally by: DaDutchDude Holy [censored] Batman! Dev Goodness Galore!
[..,]
6) Windows Server 2008 HPC: ORLY? I read somewhere about 2008 HPC and it looks like it would fit well with CCP, but can you tell us more about what you are specifically hoping to gain from this upgrade?
Looking forward to your responses!
To answer my own question (or at least guess for an answer): From http://go.microsoft.com/fwlink/?LinkID=121218 , I found some interesting info:
Quote: The Microsoft Message Passing Interface (MS-MPI) standard is a portable, flexible, vendor- and platform-independent standard for messaging within and between HPC nodes. MPI is the specification, and MS-MPI, MPICH2, and others are the implementations of that standard. MPI2 is an extension of the original MPI specification. MS-MPI is a messaging interface based on the Argonne National Laboratory open-source MPI2 implementation and is compatible with the MPICH2 reference implementation and other MPI implementations.
Fundamentally, MPI is the application interconnection between nodes on an HPC cluster, tying nodes together. MPI provides a portable and powerful interprocess communication mechanism that simplifies some of the complexities of communication between hundreds or even thousands of processors working in parallel.
The MS-MPI in Windows HPC Server 2008 uses a new RDMA-based NetworkDirect interface for best performance and CPU efficiency. As shown in Figure 8, NetworkDirect uses a more direct path to support networking hardware, providing extremely fast and efficient networking. Speeds and latencies are of the same order as custom, hardware-specific, high-speed MPI drivers from hardware providers.
This smells like the Infiniband project has been cooking in the testbed-oven on Windows HPC Server 2008 all along, using these features to speed up the infrastructure and eventually be the base for more infrastructure-independent code that allows for more scalability.
Of course there's a bunch of other improvements, like a new TCP/IP stack which should help with connections to our clients, a whole host of features that will make the server administrators happy since it'll make server deployment and management easier and also some tools that might help developers debug stuff in the complex infrastructure. So all in all, this looks like quite a big step, even though the immediate differences to us users will most likely be negligible.
So devs, how's my guesstimate?
|
Vyktor Abyss
IONSTAR Curatores Veritatis Alliance
|
Posted - 2008.09.29 02:27:00 -
[136]
Just want to add, I'm no suck up or fanboi and have been very frustrated by the lack of CCP presence on the forums recently, but I have nothing but praise for you CCP Explorer for your cander in this thread.
Top Blogging! Keep it up!
|
PingKiller
Caldari Deep Core Mining Inc.
|
Posted - 2008.09.29 03:46:00 -
[137]
Edited by: PingKiller on 29/09/2008 03:47:23
Originally by: CCP Explorer
Originally by: Chainsaw Plankton
Quote: Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and four other solar systems are assigned to that pool.
heh what are the 4 other systems amarr, motsu, rens, orsulate (or whatever that smelly frenchie system is called)?
The only market hub on a dedicated machine is Jita, the other systems are missions-running systems such as Motsu and Saila.
Wouldnt be bad if u check out mission hubs like Dodixie, Aunia + Auvergne from Sinq region as well... In times i tested Motsu, Dodixie (and neighbour systems) had equal "lag" before StacklessIO came in, so dunno about now...
|
Ashlee Darksky
Minmatar Forum Insurgency
|
Posted - 2008.09.29 09:38:00 -
[138]
This is freaking epic! Actually, it's far beyond that but I have no other way to describe it.
I do have a question though... If a stack falls over, does it become a heap?
Combat: You dev team perfectly strikes lag wrecking for epic improvements!
Thank you devs
---
> I see fail everywhere, and it's like they don't even know they're failing > Bring me the heads of 10 carebears, 5 bottles of BBQ sauce, firewood and a box of matches!
|
Kaeten
Hybrid Syndicate
|
Posted - 2008.09.29 09:47:00 -
[139]
nice work guys, might have a chance after all ________________________ I'M POOR
|
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.29 12:59:00 -
[140]
Edited by: Bartholomeus Crane on 29/09/2008 12:59:24
Originally by: Faffywaffy
Originally by: Bartholomeus Crane In short, IO requests not-in-order, game-logic in-order.
That is the desired effect, but the blurb on the dev blog seems to imply that the game logic handling layer can process requests out of order. If it were only the I/O layer that could read requests out-of-order, it could not possibly "manifest itself in strange client behaviour as requests were delayed and completed by the server much out-of-order."
I read that statement as "manifest itself in strange client behaviour as (IO) requests were delayed and completed by the server (i.e the network layer) much out-of-order." As such, to me, it was nowhere 'implied' that game-logic is handled out-of-order, and that was confirmed here. Feel free to continue whipping, but to me, this horse is well and truly dead. -- Quis custodiet ipsos custodes? |
|
Cefany
|
Posted - 2008.09.29 16:17:00 -
[141]
Were there any improvements made to anything layer 5 and below? I'm not sure if this falls in the scope of what the StacklessIO project, but were there TCP optimizations included?
|
Aselus
Amarr Crimson Right
|
Posted - 2008.09.29 16:47:00 -
[142]
good job guys, never worked that much w/ netcode, but stackless/parallel paradigms are always awesome! power to the people! errr coders! ~~~~~~~~~~~~~~~~~~ ~Eternal Swordsman
Amarr Victor!
|
Letrange
Minmatar 17th Minmatar Tactical Wing
|
Posted - 2008.09.29 17:21:00 -
[143]
*geekgasm*
You know, at the end of the day, even with all the problems that are still there to be resolved, it's a willingness to discuss tech issues like this that REALLY puts CCP in a different league than other MMO developers/publisher. As much as we all sometimes complain about either Lag or the harshness of eve, it still offers one of the most unique MMO experiences going.
I await with impatience the point in time when all the current round of upgrading finishes and Tranquility gets certified somewhere on the list of supercomputers in the world (any idea where we currently stand?).
|
SillyWaif
Galactic Kingdom
|
Posted - 2008.09.29 18:25:00 -
[144]
Great blog, and awesome to see the interaction with the Devs in this thread. Greatly appreciated |
Elaron
Jericho Fraction The Star Fraction
|
Posted - 2008.09.29 18:34:00 -
[145]
It's good to read that Stackless IO is coming to clients as well but would it be possible to elaborate how it benefits the client-side in particular? |
Pantheon Lea
Farmer Boyz
|
Posted - 2008.09.29 18:45:00 -
[146]
Normally Jita reaches a maximum of about 800-900 pilots on Sundays. On the Friday following the deployment of StacklessIO there were close to 1,000 concurrent pilots in Jita and on Saturday the maximum number reached 1,400. This is more than have ever been in Jita at the same time.
Why do i always have to read between the lines
- There was always arround 500-600 people stuck in Jita at any given time...
Anyways, what i love about CCP is that they just say it out loud, imo that deserve only hurrays for using their time improving instead of denying.
Stackless IO, Infiniband testing and HPC hardware, things like that gives reading old devblogs and patch notes a whole new dimmention.
I don't play much anymore, but i love to be updated about how EVE is growing.
|
Fink Angel
Caldari The Merry Men
|
Posted - 2008.09.29 22:19:00 -
[147]
Excellent feedback here. Thank you Dev Team. |
Baredil
Real Nice And Laidback Corporation
|
Posted - 2008.09.29 22:45:00 -
[148]
Great stuff guys!
Now, if y'all can just get it all ported over to AIX or Solaris or something ...
O/S flame war in 3...2...
|
Rashmika Clavain
Gallente Revelation Space
|
Posted - 2008.09.30 11:35:00 -
[149]
So have CCP created a new network protocol here or have I misunderstood how StacklessIO works? Removed. Please keep your EVE signature related to your EVE persona and not that of a real life politician. Navigator |
Moranti Patron
|
Posted - 2008.09.30 12:57:00 -
[150]
Just a quick comment.
Thank you.
Every step you take makes our fun that much more.
Now sell some of the tech, and make some extra money, and lower our fees... Love ya Devs!
|
|
Tashia Rizti
Lost Terra Technology
|
Posted - 2008.09.30 13:37:00 -
[151]
I don't normally post on dev blogs, but it was nice to see some information on what is being done on the tech side. I Find it very interesting (and so does many others it seems!) keep them coming please! |
Bartholomeus Crane
Gallente The Crane Family
|
Posted - 2008.09.30 14:39:00 -
[152]
Originally by: Rashmika Clavain So have CCP created a new network protocol here or have I misunderstood how StacklessIO works?
Define network protocol?
Seriously though, I think CCP developed a new and better way of using the existing network protocols they use. Your client is still talking to the server though TCP/IP. -- Quis custodiet ipsos custodes? |
Haniblecter Teg
F.R.E.E. Explorer Elitist Cowards
|
Posted - 2008.09.30 16:53:00 -
[153]
I think its CCP's transperancy that really turns me on to them. Some claim otherwise, but they've just been spoiled by CCP, and gruff when it appears they're not being as forthwith.
Hats off no matter. Love to see how these future improvements will translate to fleet battles. |
Washu
Caldari Indigo Rain
|
Posted - 2008.10.01 16:59:00 -
[154]
Originally by: porkbelly
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are.
Out of curiosity, were you using IOCP previously, but the method in which it interacted with stackless python was inefficient for greater loads, or did you use an older method, such as polling/asynchronous sockets?
|
Lady Gwynneth
|
Posted - 2008.10.02 02:22:00 -
[155]
Originally by: CCP Explorer Edited by: CCP Explorer on 27/09/2008 17:48:01
Originally by: Levitikon If you have time to answer silly questions on the forums, could you please be so kind and reassign a bit faster node (hint; Jita; hint) to MSHD solarsystem? We already had three node crashes in last 30 minutes, fourth won't make much difference.
The system administrators have been monitoring this fight closely. MSHD-4 has now been remapped to one of the dedicated machines (native 64 bit code and Jita-style hardware).
Hi CCP Explorer, nice to hear about the improvements and thanks for this great dev blog.
I see that you are using windows servers.
Did you ever make some tests how your code performs using another operating system or even server architecture ?
If your application is multi threaded you might also have a look at some server/cpu designs that support this very good and which also have advantages regarding low energy consumption which might also serve you well in the hosting center to save energy costs (Sun T5220, ...) ?
Or did you try Open Source Unix systems that scale very good under load and also offer a very well network stack ? FreeBSD at the moment seem to scale very well regarding multi CPU/core systems (8 / 16 CPUs).
Linux of course might also serve you well.
I simply wonder if you might get some additional performance gain by this as Windows server in the past very often were limited regarding top notch performance and not always make best use of server hardware.
|
Andreis
|
Posted - 2008.10.02 02:41:00 -
[156]
After reading the blog on it I call this complete Epic Win. Faster smarter better and nothing was broken in the process. Total victory \o/
|
XMM1
|
Posted - 2008.10.02 03:44:00 -
[157]
Epic victory or a sad end of the long story? On October 23, 2006 one of your developers wrote: "I currently maintain Stackless for what its worth and I also use it in my day to day work at CCP." See here And they start using anisochronous sockets!! Brilliant!
|
adriaans
Amarr Ankaa.
|
Posted - 2008.10.02 09:21:00 -
[158]
yay love these kind of blogs :D
seems to be working rather well according to you're data :D although some other systems with merely hundred players in have been 'lagging' worse than old jita did sometimes (half of local kept commenting on it, so doubt it is/was a local issue)
ohh and looking forward to full 64 bit :D -sig-
Support the introduction of Blaze M crystals for Amarr!
|
Jolliejoe
Caldari
|
Posted - 2008.10.02 12:01:00 -
[159]
So your stackless IO thing didn't improve anything... Anything more you can pull out of your sleeves?
|
SFShootme
Reikoku Band of Brothers
|
Posted - 2008.10.02 14:51:00 -
[160]
Yah! Blogs 4tw..
In m-o the other day the node was handeling the 1000+ in local quite nicely, untill the node itself crashed. Now the crashing really wasen't a problem. But logging in took a VERY long time. (4 hours in some cases)
If you could somehow give more priority to people logging into said node (after a crash) it would make nodecrashes alot less of a pain.
[VIDEO] Paroxysm
|
|
|
CCP Explorer
|
Posted - 2008.10.03 18:18:00 -
[161]
Edited by: CCP Explorer on 03/10/2008 18:23:59
Originally by: Lady Gwynneth I see that you are using windows servers. Did you ever make some tests how your code performs using another operating system or even server architecture?
We haven't made comparisons on using Windows vs. other operating systems. The decision to use Windows was in part based on convenience. Since the entire application suite runs on Windows then developers can run a server, proxy and clients on their local workstation, which simplifies and eases development. In part the decision was also based on the technical support we get from IBM, our hardware vendor and Microsoft. In addition to Windows then we also use Microsoft SQL Server so if we encounter any database issues then there is no vendor finger pointing.
There are pros and cons to different operating systems but the homogeneous operating environment helps in keeping EVE development agile.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.10.03 18:34:00 -
[162]
Originally by: XMM1 Epic victory or a sad end of the long story? On October 23, 2006 one of your developers wrote: "I currently maintain Stackless for what its worth and I also use it in my day to day work at CCP." See hereAnd they start using anisochronous sockets!! Brilliant!
I'm not sure if I understand your post, but the developer posting there, CCP lickspittle (Richard M. Tew), is referring to the fact that he is one of the maintainers of Stackless Python.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
Eraggan Sadarr
Phoenix Tribe
|
Posted - 2008.10.04 11:09:00 -
[163]
Originally by: CCP Explorer
Originally by: Reptzo Are you guys switching all coding over to native 64 bit? Is it already switched?
We did a lot of work this week on 64 bit code and Tranquility is currently running a few nodes with all native 64 bit code.
Cool, Now we are talking :) Congratulations on a milestone upgrade.
Eve Market Scanner |
BillyBong2
Amarr Obsidian Inc. KIA Alliance
|
Posted - 2008.10.06 14:59:00 -
[164]
So since the stackless has been introduced how many node crashes have their been? I know of two obviously, just wondering if there are any others. |
Ydyp Ieva
Caldari Amarrian Retribution
|
Posted - 2008.10.08 17:25:00 -
[165]
If stacklessIO is already deployed then I have to say that it caused me to get a 3-4 second response on the warp command. Weird thing is that everything else is pretty responsive. FPS stays high up, modules react on the moment I activate them, but oh dear if try to warp, then it can be that I have to wait 3-4 seconds before I even hear the sound notification of warping and see my ship turn to align.
|
Slistine Death
Sons of Angelus Mortis
|
Posted - 2008.10.11 00:45:00 -
[166]
Good job CCP. Thank you for finally taking a step to reduce "lag" and "features".
|
davcin
Caldari davcin Corp
|
Posted - 2008.10.11 18:54:00 -
[167]
StacklessIO was deployed.
Did it helped? Yes it did.
But it did bring problems. Problems like: - missiles launchers sometimes don't reload - a part of the ammo sometimes disappears from missile launchers bays - drones sometimes are taking ages to deploy - drones sometimes are not launched correctly (you have 7 in drone bay, can only control 5, sometimes 6 are launched leaving one drone inactive in space) - warp 3-7 sec initiation delay (as said by some before) - drones take some time to follow a attack order (in system with lag) - moving modules between hangars (ship/station/etc) takes ages, more then was taking before in heavy lagging systems - some systems still lag all over the place (like Ruvas) - open cargo of wrecks is still taking ages (in system with lag)
StacklessIO may have improved allot of things but problems remain and it isn't the 'miracle' you keep advertising.
|
GLEV
|
Posted - 2008.10.13 14:23:00 -
[168]
Great work Guys.I have a question to ask.isn't windows server 2003 and sql server awe enable applications to use more ram than 4Gb.If so why we need 64bit code?when you could just upgrate ram on the 32bit win server and just enable awe. Thanks in advance. Keep up the good work:)
|
Captain Nuf
Eve University Ivy League
|
Posted - 2008.10.15 17:13:00 -
[169]
I would like to nominate the persons responsible for this for a nice beer bonus.
The Jita/fleet combat lag issue has been one of my longest standing and biggest Eve frustrations that has several times tempted me to sadly consider checking out other MMOs.
Nice save - keep up the good work in this regard. I've noticed. CNUF-
|
Galmar Grief
Caldari Giants in the Playground
|
Posted - 2008.10.17 11:32:00 -
[170]
The pictures of the original post.. dead links.
- - - UI Suggestion for Missiles - Can we please turn the damn things off / disable the shake? |
|
Tina OPPENHEIIMER
Minmatar New Liberty FR
|
Posted - 2009.01.10 15:43:00 -
[171]
Too many lagggg !! It's impossible to play !! The system Pator and others are out of control !
I will leave Eve and say byebye !!
|
Stonecold Tyrannosaurus
|
Posted - 2009.02.14 11:37:00 -
[172]
I'm curious about how to read the graphs. Are they displaying felt lag? In the second lag graph, for example, were some players experiencing a round-trip time of ten seconds at 20:34? Were players experiencing a round-trip time of half a second or so during the rest of the day?
|
Ipooprai Nbows
|
Posted - 2009.04.08 18:25:00 -
[173]
Edited by: I****rai Nbows on 08/04/2009 18:28:34 I read the presentation you guys did at pycon this year, and was pretty impressed with StacklessIO. It mentioned that you were open sourcing StacklessIO; is it publicly available yet?
EDIT: Just realized that presentation was a few days ago and it says it'll be OSS'd in a few weeks. Can't wait to take a look at it.
|
|
|
|
Pages: 1 2 3 4 5 6 :: [one page] |