I have a concern. As you point out, this system means that ajacent systems tend to share the same physical hardware.

Doesn't that massively increase the chances of spillover tidi in nullsec battles: i.e. our staging system is overloaded so every system next to it is overloaded, massively increasing the pain in the ass to leave? Or the system the fight is in is massively overloaded, causing all systems around it to be massively overloaded as well making it a giant pain in the ass to get there?

I have a concern. As you point out, this system means that ajacent systems tend to share the same physical hardware.

Doesn't that massively increase the chances of spillover tidi in nullsec battles: i.e. our staging system is overloaded so every system next to it is overloaded, massively increasing the pain in the ass to leave? Or the system the fight is in is massively overloaded, causing all systems around it to be massively overloaded as well making it a giant pain in the ass to get there?

If your fighting system is reinforced that shouldn't be an issue. This is also no more of an issue than it used to be, and now the TiDi you create in your staging systems will at least not be affecting players on the other side of the universe who have nothing to do with your pew pew.

And of course more nullsec nodes mean smaller pockets grouped together.

[quote=Weaselior]and now the TiDi you create in your staging systems will at least not be affecting players on the other side of the universe who have nothing to do with your pew pew.

If your fighting system is reinforced that shouldn't be an issue. This is also no more of an issue than it used to be, and now the TiDi you create in your staging systems will at least not be affecting players on the other side of the universe who have nothing to do with your pew pew.

If your fighting system is reinforced that shouldn't be an issue. This is also no more of an issue than it used to be, and now the TiDi you create in your staging systems will at least not be affecting players on the other side of the universe who have nothing to do with your pew pew.

And of course more nullsec nodes mean smaller pockets grouped together.

Unrelated question for Prism X:
Why is it that the Empire nodes have so much more peak traffic than the nullsec systems? Is this to compensate for the occasional extreme swings in nullsec load (huge unexpected fights) that have no highsec equivalent, or something else?

The bolded part concerns me. It makes a certain amount of sense to avoid inconveniencing the three people in some system in Solidude for reasons they don't understand, sure. However it also makes a great deal of sense when you have a cluster of systems expected to be used heavily to offload the issues caused in those systems to unused systems in buttfuck nowhere as a sort of heat sink - it seems to me that it is better to be inconveniencing three ratters in Solitude because of 1000 people in nullsec than doubly inconveniencing those 1000 people in nullsec.

I don't mean to criticize the work you've done here - this all looks extremely impressive. I'm just concerned that it's assumptions on nullsec battles are not correct and may wind up creating more inconvenience overall in certain situations.

afaik from the dev blog it wont work exactly as you would say. if your staging close to a hostiles staging then upon downtime when the server runs a new premapping it may identify the staging systems as load hotspots and could well at the last splitting to discrete servers decide to split your staging and the hostiles staging to different servers.

if u plan on changing alliance / coalition staging once every day after downtime then i guess you could still grief in your original way.

Prism... the pre-loader to split the cluster load runs every downtime right?

Wormholes do not need to be split by proximity. There's no sense of locality or proximity in WH space so they just get a very dumb but efficient method applied to them.

However I reduced the number of nodes running WH space. We've added a few back, apparently I should look at adding more before the weekend.

Sorry

Edit: Yes this distribution is remade every startup.

Wormholes do not need to be split by proximity. There's no sense of locality or proximity in WH space so they just get a very dumb but efficient method applied to them.

However I reduced the number of nodes running WH space. We've added a few back, apparently I should look at adding more before the weekend.

Sorry

Edit: Yes this distribution is remade every startup.

Looks like a whole load of baloney! Us lowsec residents have been suffering from utterly ridiculous amounts of tidi, in sub-100 man fights or even moving thirty people ten gates. If you did do this rebalancing act, you sure seemed to have forgotten the systems between 0.5 and 0.0.

We're hoping to have this code out by tomorrow Wednesday, December 4th.

So I'm going to quote you so you can't edit out the part where you look silly, and then quote the devblog that explains why you look silly.

Looks like a whole load of baloney! Us lowsec residents have been suffering from utterly ridiculous amounts of tidi, in sub-100 man fights or even moving thirty people ten gates. If you did do this rebalancing act, you sure seemed to have forgotten the systems between 0.5 and 0.0.

How is this being applied to wh systems?

It isn't. Solarsystems that are not geographically connected (WH systems) and non-solarsystem services are loadbalanced differently.

So does this mean that there are systems that are non-wh, non-solar systems? This further satisfies my theory that our new systems could be in the folds between known space in the dangerous pockets of spacetime like "The Vapor Sea".

also just saying that the phrase used being 'empire systems' would most likely refer to both high and lowsec systems, as im fairly certain all of the 4 main factions have lowsec areas. if you consider the area u fly in as being non-empire i suggest looking at the logos on the station toolbar next time you're docked up.

So does this mean that there are systems that are non-wh, non-solar systems?

Prism is it possible to put hard barriers for the pre-loader to stop connecting low and null sec systems together on one node? does this happen already?

when remapping systems from an overloaded node to another can a low and null system end up on the same node after tomorrow?

Currently Nullsec is split off from Empire Space (High and Low). That's a change we did a long time ago and was covered in my first draft of this blog. That was however about three times the size of this one, without pictures. So yeah we could so that, but we don't.

What's the reasoning behind this?

they dont have to deal with the load bearing requirements of hundreds of fighter bombers so im sure they'll be fine Prism!

Given that you know that inter node jumps are the best because you split the work between two nodes, wouldn't it be better to work with node pairs. Apply the exact same algorithm to break down all the systems into equal size groups, but then right at the end take the group of systems and split them across a pair of nodes so as to create as many inter node jumps as possible.

Pretty sure those 3000 sentries from 600 Domis do not create server load.

Well, for one, Dogma doesn't need to calculate anything about the movement of sentries, and notifications about their movements to any of the clients.

That's starting to sound like a Colour problem. Ultimately you want to maximise the number of local nodes without them touching themselves, while minimising the global distribution of any individual colour. This does the best to maximise the high-performance transitions while keeping TiDi relatively local to its cause.

Well, for one, Dogma doesn't need to calculate anything about the movement of sentries, and notifications about their movements to any of the clients.

While FBs require AI (ok, you can laugh), physics and position notifications to everyone in grid.

I'd say a totally different workload.

when was this fixed deployed or is it still in the pipeline to release?

I am not exactly sure why would you say that. Perhaps you think that sentries don't move? They do.

That's simply not true. I can pull up an overview with sentries on it, and they move, at 1m/s. That means the server has to update me constantly with the positions of all those sentries, same as it does with fighterbombers or any other drone.

So I'm going to quote you so you can't edit out the part where you look silly, and then quote the devblog that explains why you look silly.

Baloney!

Yes, empire refers to high and low security space together.

Ok, write me the equation of linear, uniform movement. And tell me about predictability of such movement over time, which is a key part of client-server protocol as of right now.

Then the equation of uniformly accelerated motion (aka orbit) -- which can change at owner's whim, anytime.

Sure they move.

Sure the workload is totally different.

QED.

I am slightly confused. You started the devblog by asserting that the old system was bad, because it put clusters of connected systems on one node (and intra-node jumps are more expensive than inter-node jumps). Yet you end up with a system that puts even bigger clusters of connected systems on the same node? From the pictures it looks like instead of constellations, now huge chunks of regions share the same node. Wouldn't that lead to even more intra-node jumps?

You seem to know quite a lot about the EVE server/client internals. Are you sure you didn't want to post this on your CCP character?

As others have already said, the new allocation could lead to even higher load being put on the nodes in battles, as it seems now that any sort of reinforcements or auxiliary fights will directly contribute to the load of the fleet fight node itself. And this will also happen with a pre-reinforced node - even if you take the contested system out of the picture, all the surrounding area still shares the same node. Whereas in the old distribution, it is rare to even find a system whose direct neighbors are all on the same node.

Gameplay-wise, it seems that the vast majority of people use their groups of drones as a single unit,
which means that all groups of drones could behave as a single unit,
which also means that the server would only need to do calculations for a single unit.

i don't know how they reinforce nodes, but they could just take a single system out of the node and put it on a stronger node. They don't have to replace the whole node with all its systems, only pick a single one.

That is exactly what reinforcing a system means.

Hello everybody, my name is CCP Prism X and I am a Libra!

The bolded part concerns me. It makes a certain amount of sense to avoid inconveniencing the three people in some system in Solidude for reasons they don't understand, sure. However it also makes a great deal of sense when you have a cluster of systems expected to be used heavily to offload the issues caused in those systems to unused systems in buttfuck nowhere as a sort of heat sink - it seems to me that it is better to be inconveniencing three ratters in Solitude because of 1000 people in nullsec than doubly inconveniencing those 1000 people in nullsec.

I don't mean to criticize the work you've done here - this all looks extremely impressive. I'm just concerned that it's assumptions on nullsec battles are not correct and may wind up creating more inconvenience overall in certain situations.

So, uh, how do you propose shooting and killing drones should work?

Anyway why couldnt you just have a node(s) per region?

I am slightly confused. You started the devblog by asserting that the old system was bad, because it put clusters of connected systems on one node (and intra-node jumps are more expensive than inter-node jumps). Yet you end up with a system that puts even bigger clusters of connected systems on the same node? From the pictures it looks like instead of constellations, now huge chunks of regions share the same node. Wouldn't that lead to even more intra-node jumps?

when was this fixed deployed or is it still in the pipeline to release?

There's no sense of locality or proximity in WH space so they just get a very dumb but efficient method applied to them.

I hear this guy listens to really weird music (NSFW) which is probably indicative of his cognitive capacity. Probably not worth reading this!

One more thing to mention, as a part of this change then the cluster is starting up 2 minutes faster. See here https://forums.eveonline.com/default.aspx?g=posts&m=3897467#post3897467 and here https://forums.eveonline.com/default.aspx?g=posts&m=3899297#post3899297 for details.

How is work progressing with the dynamic node reinforcement where you move sols between nodes while players are busy in those sols?

.... snip.....

The final piece of this puzzle, the intra-node jumps vs. inter-node jumps we ultimately want to solve with Brain in a Box.

Would it be possible to do something like this?:

- freeze game (TIDI 100%) in old node
- spawn new node with 100% TiDi
- Show players info about load balancing or something to let them know they are changing servers
- move people to new node with all player/ship states untouched and make it take as long as it is needed
- all new player system entrences during the node movement is put too new node
- after all players are moved OR certain timer have passed despawn old node and set TiDi to appropriate level on new node

If TiDi is good enough for blobbers in Null Sec is should be OK for empire people. When a node goes in to TiDi when its reinforced and there are less people than are in Jita, Something is wrong, Very wrong. When we have 60 people in a fleet and we undock, we get TiDi, when we swap ships, TiDi, when we jump through a gate, TiDi. None of thses should cause TiDi, yet they do.

TiDi was introduced as a bandaid while CCP was working on a Fix. Now you are promoting TiDi as the fix.

So what is it?

Is TiDi the end all fix or is CCP working on fixing the core issues that cause TiDi to begin with?

If TiDi is the fix, you are doing it wrong.

I hear this guy listens to really weird music (NSFW) which is probably indicative of his cognitive capacity. Probably not worth reading this!

wtf

from what i can understand i think he means the opposite... cut up the load balancing down to the final level then split the final level across 2 nodes ensuring maximum contact surface between the 2 servers.

instead of creating a line between the localised cluster of systems you create a chessboard style distribution to maximise the amount of server to server jumps rather than intra server jumps (that are considered inferior to discrete server to server jumps)

you have the localisation at a cluster wide scale, then when u get to the singular node scale it switches to a high diffusion model. the problem would be how to keep the remapping from morphing the diffused local distribution from becoming localised through time.

So we have like, what? 6 - 8 minutes of sleep left daily?

Read the Devblog and must say, looks like you guys out in a lot of work to make TiDi less frustrating, thank you a lot for that :)

Is it possible for you to make an addition to the Devblog explaining how the load is distributed in w-space by chance? We now know and understand in detail what is happening in known space, but it could be very cool to know how you solve w-space, even though it might be "dumb but efficient" :)

So we have like, what? 6 - 8 minutes of sleep left daily?

We're still waiting for that dev blog mate

but my understanding is that you are still trying to balance the load geographically instead of doing it statistically?

First of all, should you actually stop treating the problem for nullsec and highsec (+lowsec I guess) as similiar?

Related to the randomness of the nullsec load, my second concern is how fast you will iterate this node splitting or will it be set in stone and forgotten about in few months?

Wont this mean that hosts running the servers of nullsec regions of north are almost idling, supporting the occasional ratter and which ever host gets systems from current conflict regions (Immensea etc) is always going to be heavily overloaded?

Can we get a copy of that map with System names instead of IDs?

Would it be possible to have a node set the TiDi to 0 to permit a hot node remap without kicking out the players?

First of all, should you actually stop treating the problem for nullsec and highsec (+lowsec I guess) as similiar? Because the way I see it, highsec system load should be rather predictable and mostly consistent due to tradehubs and missioning centers remaining largely the same. Nullsec however is rather more unpredictable, system load is generated by whim of player coalitions and where they choose to clash that day (timers and random acts of violence).

If all you want it for is a large star map, with system names: http://imgur.com/a/opwDm

(use the cog in the top left of each image to get the full resolution version)

Absolutely not. This is a static load balancer that balances system between nodes at server startup. I'm actually reading a paper from some people at the University of Bonn about predicting destinations based on previous system jumps. It's pretty interesting but my brain now hurts. But if we could hook something like that in we could detect staging systems forming. We'd still not detect sudden escalations. Sudden escalations need dynamic load balancing if we're to handle them gracefully under the current CPU per User fingerprint.

..............We'll need to reduce the CPU footprint per user if we want to prevent Fleet TiDi (and my money is on that only increasing fleet sizes until TiDi becomes unbearable again).

So what efforts are being done to reduce the CPU footprint per user?

I've mentioned it elsewhere, but why are these machines not virtualised (or are they?) surely something like vMotion would be able to move high-use systems onto dedicated hardware without the need to pause anything.

Ive heard the words "Brain in a Box" quite a bit and seen vague descriptions of it having to do with preparing session change data on a separate node. But is there a full description somewhere? What it does, how much load it will remove, and so on?

I've mentioned it elsewhere, but why are these machines not virtualised (or are they?) surely something like vMotion would be able to move high-use systems onto dedicated hardware without the need to pause anything.

Ive heard the words "Brain in a Box" quite a bit and seen vague descriptions of it having to do with preparing session change data on a separate node. But is there a full description somewhere? What it does, how much load it will remove, and so on?

@ Prism X:
Over what timeframe are the cpu metrics used for premapping collected? Could a single escalated fight in a usually empty system not skew the metrics, forcing the system to become reinforced the next day? Are outlier datapoints stripped?

Check out some math and numbers that make me feel dumb, and make the server feel great with CCP Prism X's new Dev Blog.

... because currently TiDi kills the most of fun in epic battles.

It's meant to offload work currently being done by solar system nodes onto a different node, as well as to reduce the total amount of work needing to be done over and over again at certain time. How much it will offload is wholly dependant on the end result and how much of the current code is refactored into C.

I'm not sure if there are external sources with information for you. If I were at the office I could look something up for you or ask people. But I'm not. Perhaps I'll remember tomorrow!

No. TiDi resuscitates fleet fight fun, keeping it alive but not entirely healthy.

What we had before ritualistically pillaged and dismembered fleet fight fun.

Check out some math and numbers that make me feel dumb, and make the server feel great with CCP Prism X's new Dev Blog.

is there any kind of support for university papers ? i might actually be interested (not on a technical level, but for markets or politics)

Ive heard the words "Brain in a Box" quite a bit and seen vague descriptions of it having to do with preparing session change data on a separate node. But is there a full description somewhere? What it does, how much load it will remove, and so on?

You really think a company, running the largest gaming cluster for over 10 years now would not have already bought a stock solution if it worked for them? Cute.

As far as the metagame is concerned, even without a published node map it's going to be exploited. For example in a defensive Sov war, if most of a region is on the same node it's not going to be hard to find a linked system by trial and error and dock/undock repeatedly to cascade the entire node (most of a region in the current scheme) into a sustained 10% tidi to discourage siege fleets from grinding structures.

"Stock Solution"?

I'm not sure you appreciate the complexity it would take to roll out such a massive deploy and use live migration all while pumping the entire local network through virtual switches.

Easy it isn't...

Environment integration and live rollout of a technology have nothing to do with it. First that technology has to solve your particular problem. And yes, VMware ESX / vMotion "live migration" is pretty much a stock solution as far as virtualization goes.

TQ "nodes" are not machines, they are processes. With each process serving multiple solar systems. For that reason, you can't move a high-use solar system by moving a virtual OS around.

As far as workarounds go, one could run one virtual server with a single process running a single solar system for every system, sure. With all the increased overhead that brings along with it. And I am equally sure some Dev at CCP evaluated that already. And the fact that they did not adopt it (or something similar) means it did not work for them.

And if all that is sorted out, the EVE server code runs at 1HZ. Freezing a node for 2 seconds to copy it over to somewhere else are 2 missed server cycles that the clients have to resynchronize with again.
Which means a major redesign and rewrite of the network code on the client and server side.

For those kind of development resources, you need a really strong business case. And while dynamically reinforcing nodes is a nice target, we push those into TiDi as well. All the time. Which means it's essentially a band-aid. Not something you get a man-year of development effort approved for.

The underlying VM has no idea at all it's been moved.

And since it's taken a nontrivial amount of time to move relative to the 1HZ physics engine, meaning that the odds are very good that your half a second will cross a tick boundary, that means that every move must be followed by a resync with adjacent systems to get everyone back on the same page, right? If one node is off by a server tick, how do you handle that?

And since it's taken a nontrivial amount of time to move relative to the 1HZ physics engine, meaning that the odds are very good that your half a second will cross a tick boundary, that means that every move must be followed by a resync with adjacent systems to get everyone back on the same page, right? If one node is off by a server tick, how do you handle that?

Vmotion at least is truly transparent to the underlying VM. You will see a "pause" but incoming network packets etc. are not dropped ,just queued while the machine is in motion afaik.

During TiDi different systems are not running in sync either.

(I'm not saying this as a proof that this will be easy, rather as anecdotal evidence for it.)

Nope.

Set up a continuous ping of a VM, then vmotion it, and you'll see a couple of dropped packets.

Ping is connectionless and has a timeout of 3 seconds.

A TCP connection is - duh! - connection based, and usually the timeout is at 30 seconds.

Perfect? No.

But a dropped ping doesn't necessarily mean a dropped connection.

Maybe I'm misunderstanding something, but as far as I can tell this will only have a negative effect on the quality of game play in regards to already painful fleet combat.

Frankly I don't remember the last time I was in a full fleet and there wasn't heavy Ti-Di. Every time a solitary 250 man fleet jumps a gate the system spikes to 10% tidi for 30-45 seconds. Even if every fleet fight was on an individual reinforced node (reinforced nodes are the exception, not the rule) the issue of gate Tidi is going to be exponentially worse under the new regional scheme since every individual fleet in the area traveling to (or from) the combat system is going to be sequentially triggering gate lag on the same node. It's going to be a particularly painful change given the recent quality of life hits to the majority of fleet ships, there's nothing fun or engaging about staring at a warp tunnel for 10 minutes per system the entire trip home.

As far as the metagame is concerned, even without a published node map it's going to be exploited. For example in a defensive Sov war, if most of a region is on the same node it's not going to be hard to find a linked system by trial and error and dock/undock repeatedly to cascade the entire node (most of a region in the current scheme) into a sustained 10% tidi to discourage siege fleets from grinding structures.

Yes the old system wasn't perfect, but the guy ratting in an empty system halfway across EvE could have just moved over to a different system and continued ratting. Maybe this is the right solution for Empire where loads are usually steady from day to day but it's the wrong approach in Nullsec.

still waiting for confirmation that failed wormhole jumps with traffic control messages count against the wormhole mass, but will be looked into. (meanwhile there will be support tickets, handled by uninformed customer service staff)

I could go track down an engineer if you want a totally firm answer, but my understanding is that the client is waiting for the server when jumping rather than vice versa. It's only when the *server* has finished the transfer and told the client it's OK to go ahead and load that it finishes the transaction and deducts the mass, and jump requests are not completed strictly in the order they were requested.

I dislike leaving things hanging.

Wormhole Mass reduction is well outside of the scope of my work. Obviously mass should not be decreased on a denied jump but that should be bug reported through the proper channels so that it receives proper due diligence. Apparently this has been BR'd already so YAY!

Libras are not trustworthy. We're extremely dodgy characters who seem to be willing to adopt any side of everything for the sake of whatever deviant impulses drive us at that given moment.

I do believe everything else has been covered already in previous posts.

Fly safe...
..ish!

I do believe everything else has been covered already in previous posts.

No. Distributing connected solar systems between different nodes will not make any load dissapear. It will just move it elsewhere. Perhaps I wasn't clear enough when I said that intra-jumps were more expensive: It's more expensive for that specific node as it will have to do both the tasks. Jumping between nodes does not magic away some part of the work that needs to be done. It's just done by two nodes. Remember: We do not want a fleet in the south to cause TiDi in the north.