|
Author |
Thread Statistics | Show CCP posts - 19 post(s) |
|
CCP Wrangler
|
Posted - 2008.09.26 17:35:00 -
[1]
For some time now we have been developing StacklessIO to reduce lag in EVE, and it was recently introduced on Tranquility. Explorer has written a blog detailing what this is, how it works and how it decreased lag in EVE. The blog is up for your reading pleasure: StacklessIO or: How We Reduced Lag
Wrangler Community Manager CCP Hf, EVE Online Email |
|
|
CCP Explorer
|
Posted - 2008.09.26 18:02:00 -
[2]
Originally by: Kerfira While there is no direct link to the new network layer, it seems a logical possibility that a major change like this brought a few unintended consequences. You'll probably look into this, I hope...
The Core Technology and EVE Software development teams as well as Quality Assurance and Virtual World Operations have all been monitoring Tranquility intensely since we deployed StacklessIO.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.26 18:09:00 -
[3]
Edited by: CCP Explorer on 26/09/2008 18:09:36
The old network technology limited us to using 32 bit code. We can now move to 64 bit code (we actually did the bulk of that that work this week) and can then add more memory to the cluster and make use of it. The exact plans haven't made yet so I can't tell or promise if or when that will happen. It is a possibility now.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.26 18:38:00 -
[4]
Originally by: Reptzo Are you guys switching all coding over to native 64 bit? Is it already switched?
We did a lot of work this week on 64 bit code and Tranquility is currently running a few nodes with all native 64 bit code.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Atlas
|
Posted - 2008.09.26 20:53:00 -
[5]
Edited by: CCP Atlas on 26/09/2008 20:54:46
Originally by: Syekuda Questions for the devs here: I've been thinking after I read your wonderful blog. Is the improvement only in the node or cluster where Jita is ? Also, if not which means its everywhere, will the effects (the final stats) be the same in a system which is empty and then all of a sudden, you see 300 ships using the jump gate ?
I was wondering since I don't know much about the network infrastructure that Eve uses, but is the "limitations" the same with an 0.0 system. In case you don't understand or its really not clear lets say the cluster in Jita as 32gig of ram because its always busy. Would there be the same in a 0.0 system or a system thats not normaly busy ?
This is a performance boost to all solar systems in that requests you make will reach the server more quickly. The effect is mostly noticeable where the node is heavily loaded and thus Jita has been used as an example.
This will not allow the node to process your requests more quickly though once it reaches the application layer, but you shouldn't have to wait 5 minutes for a module to activate.
|
|
|
porkbelly
|
Posted - 2008.09.26 20:58:00 -
[6]
Originally by: Bartholomeus Crane The results certainly look promising, but I would like to know what StacklessIO actually does...
Ah, yes. As the dev primarily responsible I should probably write a technical blog about it. Meanwhile IŠll offer that StacklessIO is a framework that allows us to make things such as asynchronous IO and work that is spawned off to worker threads appear as regular, blocking operations for tasklets in Stackless Python. We then use this to perform asynchronous Winsock operations using IO completion ports. The semantics are not new, but the scheduling framework and the lightweight winsock layer we use are. What it is! |
|
|
CCP Explorer
|
Posted - 2008.09.26 23:34:00 -
[7]
We did a lot of work last year in improving the client/server communication on the physics simulation. The old network technology could possibly have caused or contributed to instances of desync. StacklessIO will help in all circumstances.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.27 10:31:00 -
[8]
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied everywhere on the cluster. It is also coming to a client near you next Tuesday when EVE Online: Empyrean Age 1.1.1 will be released.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Lingorm
C C P
|
Posted - 2008.09.27 10:31:00 -
[9]
Originally by: Brock Nelson inudex, I think StacklessIO patch was applied only to nodes with heavy demands such as Jita. I understand that mission hubs have large demand but nothing compares to Jita
StacklessIO was applied to all node in the Tranquility Cluster, not just to high load nodes.
The 64bit compile of EVE has been deployed to some high load systems and we are actively monitoring the performance of these systems.
CCP Lingorm CCP Quality Assurance QA Engineering Team Leader
Originally by: Lord Fitz Eve is to WoW as Wow is to an 8 player game of Unreal Tournament.
|
|
|
CCP Explorer
|
Posted - 2008.09.27 11:03:00 -
[10]
Edited by: CCP Explorer on 27/09/2008 11:04:12
Originally by: Brock Nelson The spike using old technologies between 20:24 and 21:36, would you say that is due to the number of player during that time frame? If that's the case, how is it possible that the stacklessio maintains the average ping time regardless of the number of player in Jita? Edit: Would the average ping not correlate to the number of players in Jita?
The old technology exhibited very particular behaviour when the number of pilots crossed a certain threshold. StacklessIO provides much better performance that is less correlated to player numbers.
Quote: Also, CCP Explorer mentioned that some of the node are already running 64 bit; how does the performance of those node compare against the 32 bit nodes? I know that performance data may be biased (ie: more load in one node vs another).
It's too early to tell what the performance gain will be, we have had native 64 bit code running for only a few days now on selected nodes in the cluster. The biggest advantage is scalability, the ability to use more memory.
The normal setup in the cluster is that a blade has two 64 bit processors, 4 GB of memory and runs Window Server 2003 x64 (we are planning an upgrade to Windows HPC Server 2008). Each blade runs two nodes and each node then hosts a number of solar systems. There are also dedicated nodes for the market (each market blade runs three market nodes), dedicated nodes for corporation services, a dedicated head node for the cluster, etc. Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and three other solar systems are assigned to that pool. That pool is now running all native 64 bit code and the blades have been upgraded to 16 GB of memory.
Quote: I was wondering what exactly is causing performance issue in any node or in this case, Jita. I know its due to 1400-something people but does the demand on the node break down into something more specific? Such as people putting more people on block b/c of scams? People accessing Jita market? Players just sitting in station? Chatroom? etc
... all of the above? The Inventory System is high on the list in Jita as players move a large number of items between hangar and cargo. The Agression Manager and Damage Tracking and Calculation System are low on the list in Jita.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
|
CCP Explorer
|
Posted - 2008.09.27 16:19:00 -
[11]
Originally by: Popsikle When can we get a 64-bit client version?
Currently this is the server only. On the server there is only piece of 3rd party middle-ware for which we had to acquire a 64 bit version. On the client there are many more middle-ware components. The old network technology blocked 64 bit development so with StacklessIO there is one less hurdle for the client.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.27 17:44:00 -
[12]
Edited by: CCP Explorer on 27/09/2008 17:48:01
Originally by: Levitikon If you have time to answer silly questions on the forums, could you please be so kind and reassign a bit faster node (hint; Jita; hint) to MSHD solarsystem? We already had three node crashes in last 30 minutes, fourth won't make much difference.
The system administrators have been monitoring this fight closely. MSHD-4 has now been remapped to one of the dedicated machines (native 64 bit code and Jita-style hardware).
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.27 22:43:00 -
[13]
Originally by: Bartholomeus Crane Nice barking, wrong tree. Clearly, handling IO sequentially is not an option. IO requests were probably handled (semi-)concurrently in random order using standard IO completion port scheduling. They are used in order by the Python part by blocking threads. Problems occur when there are many IO requests at the same time. Then schedulers can get overloaded, and this is probably what happened. Anyway, there's a better scheduler now, and probably more concurrency, and IO request are handled more efficiently, so these things shouldn't happen anymore.
Exactly, the game logic was still handled in the correct order but client network request were sometimes handled very out-of-order by the server.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.27 22:49:00 -
[14]
Originally by: MotherMoon
Originally by: CCP Explorer StacklessIO was applied everywhere on the cluster. It is also coming to a client near you next Tuesday when EVE Online: Empyrean Age 1.1.1 will be released.
wait so it's not just a layer on your side but also on our side? so right now it's only on the jita side but the players in jita aren't using it? like... wait... ow
The server is now using the new network technology internally but the protocol is backward compatible so it can communicate with the clients that are still using the old technology until Tuesday.
Quote: just for tech junkies like myself could you let us know if thise upgrade to 16 GB of memory was before or after jita ran out of memory? if it was upgraded after then this is very good news, and I think you should test it out to see how many players it takes to cap out jita again!
It was upgraded after.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.28 10:46:00 -
[15]
Originally by: PC5 Are drones having any significant factor in fleet fights after that change? I think they still are making a lot of lag, am i right? Especialy in fleet fights. After that change it would be nice if you look at aspects connected to fleet fights - like POS lag, drones lag, module cycles lag. Maybe there you can improve much? Any solutions/ideas for those aspects?
Currently there is a team of developers in Need for Speed sprints measuring and looking at ways to optimise the code. One of the things they are investigating is drones/fighter vs. turrets/launchers, since it's our impression that drones/fighters contribute less to CPU cycles than turrets/launchers and we want to optimise that code.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.28 10:51:00 -
[16]
Originally by: Chainsaw Plankton
Quote: Finally there is a pool of dedicated dual-CPU machines that only run a single node per machine. Jita and four other solar systems are assigned to that pool.
heh what are the 4 other systems amarr, motsu, rens, orsulate (or whatever that smelly frenchie system is called)?
The only market hub on a dedicated machine is Jita, the other systems are missions-running systems such as Motsu and Saila.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.09.28 23:40:00 -
[17]
Edited by: CCP Explorer on 28/09/2008 23:40:44
Originally by: The Jugganaut I noticed you saying Sprints there, does that mean you work with the SCRUM development methodology?
We use Scrum for development of new features (especially where we are venturing into the unknown) and for optimisation tasks where cross-functional teams from different departments are needed. We use Staged Deliveries for more known development tasks and we when have a lot of small features (a category called "General Improvements" in the expansion plan). For defects we allocate time each week and then timebox defect work at regular intervals. The EVE Software Group is also responsible for live software maintenance of Tranquility and all live issues trump any work we are doing.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.10.03 18:18:00 -
[18]
Edited by: CCP Explorer on 03/10/2008 18:23:59
Originally by: Lady Gwynneth I see that you are using windows servers. Did you ever make some tests how your code performs using another operating system or even server architecture?
We haven't made comparisons on using Windows vs. other operating systems. The decision to use Windows was in part based on convenience. Since the entire application suite runs on Windows then developers can run a server, proxy and clients on their local workstation, which simplifies and eases development. In part the decision was also based on the technical support we get from IBM, our hardware vendor and Microsoft. In addition to Windows then we also use Microsoft SQL Server so if we encounter any database issues then there is no vendor finger pointing.
There are pros and cons to different operating systems but the homogeneous operating environment helps in keeping EVE development agile.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
CCP Explorer
|
Posted - 2008.10.03 18:34:00 -
[19]
Originally by: XMM1 Epic victory or a sad end of the long story? On October 23, 2006 one of your developers wrote: "I currently maintain Stackless for what its worth and I also use it in my day to day work at CCP." See hereAnd they start using anisochronous sockets!! Brilliant!
I'm not sure if I understand your post, but the developer posting there, CCP lickspittle (Richard M. Tew), is referring to the fact that he is one of the maintainers of Stackless Python.
Erlendur S. Thorsteinsson Software Director EVE Online, CCP Games |
|
|
|
|