| Pages: 1 [2] :: one page |
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Buyerr
   |
Posted - 2008.02.19 01:52:00 -
[31]
 Originally by: Lungorthin
I am not a technician nor a SQL / Database guru so I have to classify this as a rumor.
that is why you shouldn't ask, because without being one it would be like trying to explain how the universe works to a 6 year old child.
just accept they are doing what they can
I declare war on stupidity |
Andrue
Amarr
Federation Of Space Loonies
Culture Shock Initiative
|
Posted - 2008.02.19 09:44:00 -
[32]
Edited by: Andrue on 19/02/2008 09:44:45
The DB is not the issue. The DB will not be queried very often during a fleet fight. It will be updated (perhaps) when an item is removed or created during the game but even then it's probably on a low priority and certain items (like ammo) could be deferred until their owning container changes state or they are removed from the container. Of course ammo is stacked and the stack is likely one item with a count so even less reason to bother the DB with it.
The only lag fix that can ever work is to remove the cause. The game needs to be changed so that people tend not to congregate in one large market hub and tend not to engage in huge blob fights. Processing power and network bandwidth is a finite resource and you cannot keep expanding it forever.
--
(Sarcastic mission running veteran)
[Brackley, UK]
My budgie can say "ploppy bottom". You have been warned. |
Herio Mortis
54th Knights Templar
Dark Matter Coalition
|
Posted - 2008.02.19 11:41:00 -
[33]
I've taken a bit of interest in how the EVE cluster and clients work, as I can see the challenges in making a system of this scale work. Most of this comes from reading various dev posts here on the forum, dev blogs, stuff from stackless python mailing lists and webpage, reading logs from the logserver etc. etc.
First of all, as has been said already, there are two types of lag.
Client side lag, which is your machine not coping with the graphics and other client side stuff.
Server side lag, the cluster itself not being able to cope with the load.
For the client side stuff, much has improved with the release of trinity by offloading more work to the GPU. The client itself is basically a python runtime with 3D, sound and network libraries.
On the server side there are three main areas of the cluster. You have the database, proxy nodes and SOL nodes. When you connect, you get assigned to a proxy node. Proxy nodes handle the client connections, you stay on the same proxy node for the entire session. The SOL nodes runs one or several solar systems, handling all actions performed in that system. I also think that there are nodes dedicated to handling the market, chat windows etc.
All communication between the clients, proxy nodes and SOL nodes are performed using a custom RPC protocol that CCP wrote called MachoNet. This is their custom clustering software in essence, written in stackless python.
You, as in your ship, is represented in the software as a thread with it's corresponding data structures. When you jump to another solarsystem what happens is that this running thread gets serialized (turned into a stream of bytes) on the SOL node and shipped over to the proxy node. The proxy node then finds the destinations SOL node and ships the serialized thread to it. This node then deserializes the thread and resumes processing it. This is possible due to stackless python, it is one of the reasons behind creating stackless python in the first place.
Your ships state on the server is in a structure called godma. Godma handles all attributes, stats and other calculation on ships. It talks to a its counterpart doing the same thing on the cluster called dogma. Desync in my understanding is the result of the godma and dogma sides of the ship not sharing the same idea of things, this could happen due to differences in their programming or bugs.
The SOL nodes handle all interaction between the dogma threads on it, and this is where I believe most server lag comes from. Due to the way machonet works, they can not split a SOL nodes processing over several CPU cores, it has to run on a single core. Interaction between the threads are made easier by dividing space into grids, they only have to run the full interaction between ships on the same grid. Too many ships doing things on the same grid results in a lot of dogma threads interacting. This results in what is commonly known in module lag. When you do something like activate a module or change the ships heading, it is performed as a RPC call to the cluster, the client will start acting on it once it gets a reply back on the RPC call. If the SOL node is overworked, this response can take a good while to return.
The solution they are working on is as far as I can tell reworking machonet into something more in the line of a traditional cluster, using infiniband. This should allow SOL nodes to be spread over multiple cores but it's also a major undertaking.
 Did I write all that? Sorry for the wall of nerd-text 
-----
And *pop* went my ferox... |
1Of9
Gallente
The Circle
STYX.
|
Posted - 2008.02.19 12:12:00 -
[34]
Originally by: Herio Mortis
... stuff ...
this makes alot of sense.
|
John Ingleburn
Gallente
Maximum Freelance
|
Posted - 2008.02.19 14:52:00 -
[35]
Honestly CCP needs to upgrade its hardware again, since Trinity has came out, it has cause nothing but a problem for follow players and the like?
Ie Lagggggggg
Is there a new patch coming out for the LAG! |
William Darkk
Gallente
Garoun Investment Bank
|
Posted - 2008.02.19 18:21:00 -
[36]
The "8MB packets" thing still looks like a major issue. I'd really prefer if they sent out more but smaller packets. I don't need to know if I'm being shot at by 100 or 200 or 300 guys, I just need to know that guys there are shooting at me. Really there should be a priority on getting SOMETHING to the client asap when they enter system. Just enough information to base a decision on.
--------------------------------------------------
<3 my Drones |
Argyle Jones
Minmatar
Wreckless Abandon
|
Posted - 2008.02.19 19:55:00 -
[37]
@ Herio Mortis:
Thanks for sharing your insight. Very interesting read. Incidentally, do you know anything about the workings of the MachoNet protocol? Don't believe I've ever heard of a network protocol that sends 8MB packages before 
/Yargle
--
...Could you please rub some of that yellow powder on my lips? |
Herio Mortis
54th Knights Templar
Dark Matter Coalition
|
Posted - 2008.02.19 22:19:00 -
[38]
Originally by: Argyle Jones
@ Herio Mortis:
Thanks for sharing your insight. Very interesting read. Incidentally, do you know anything about the workings of the MachoNet protocol? Don't believe I've ever heard of a network protocol that sends 8MB packages before
/Yargle
Can't say that I know much about the inner workings of it.
As far as I can tell it looks like it is using a custom marshaller (python serializer) to save bandwidth compared to the normal python marshaller.
It is running over TCP on port 26000 for client connections, using TCP to deal with packet loss, packet ordering etc.
It's not sending 8 MB packets over the network, but some of the responses are tabular data, much like result sets from a database. Some of those can probably become rather large when warping into a grid full of fleet combat, not measured the amounts of data transferred.
-----
And *pop* went my ferox... |
Allen Ramses
Caldari
Typo Corp
|
Posted - 2008.02.20 00:13:00 -
[39]
Everything I have experienced in regards to the kind of lag observed in fleet battles has to do with network or bus saturation. These are my reasonings behind this idea. First are finding out what the problems are NOT:
1. If the problem was the database, every single player would experience the same lag.
2. If the problem was CPU cycles, mission running would be nigh impossible (it takes more calculations to run a mission than to have a comparably sized fleet battle).
3. If the problem was TQ's backbone, 200v200 battles wouldn't suffer much lag when 20k people were online.
Now, there are some times when one of the conditions above causes problems, but those are based around some kind of errata, and most (if not all) players would experience some kind of latency problem. Now for the second portion which leads me to believe the issues are network related:
1. I can run large lvl 4 missions perfectly fine solo, but in a group of 10, everyone shows minor signs of lag on a smaller lvl 3 mission.
2. When undocking/exiting warp/entering system, there is a spike on the network monitor, as new information is being broadcast to me. This happens as well when station camping blobs open fire on a war target.
3. Nature of the beast. The more active connections in a particular area, the more data has to be moved exponentially.
If one person activates a single module, the information is transmitted to 399 people. If 10 modules are activated, it's 10 times the information. And when the other 399 people activate their modules, it's 399 times more. After doing the math, that's an approximate 1.6 MILLION simultaneous initialization instructions something has to push out. That kind of throughput is on par with a DDoS.
I don't know whether it's the FSB of a machine, a switch, or a proxy to the backbone. No matter where the problem is, the problem itself seems fairly obvious. The data path is not wide enough to handle such a large movement of data. Either CCP need to invest in faster switches, wider bus, or more redundancy in their proxies.
Of course, then again, I could be wrong.
Oh, and as far as Jita goes, IIRC sharkbait or someone else (they're all the same to me :D) said that it was due to the market database being totally ripped to shreds by the end of the day. It has been widely known that 600 people in Jita right after down time don't have nearly as many issues as 300 people right before downtime.
____________________
Pimped out Raven to run level 4 missions quickly: 210 Mil ISK.
Realizing your 120 Mil ISK Drake gets the job done faster: Priceless. |
Vocis
|
Posted - 2008.02.20 03:24:00 -
[40]
Originally by: Allen Ramses
If one person activates a single module, the information is transmitted to 399 people. If 10 modules are activated, it's 10 times the information. And when the other 399 people activate their modules, it's 399 times more. After doing the math, that's an approximate 1.6 MILLION simultaneous initialization instructions something has to push out. That kind of throughput is on par with a DDoS.
If this is the case than CCPs Architects and Designers need to be sacked, immediately. The server should be pruning information that clients don't need to know about during large fleet fights.
What does it matter if my target has his hardeners on? What does it matter if his guns are on? The only thing that matters is if he is hitting me, and how much damage I am doing to him.
To send completely worthless information to so many clients should be so stupid that I refuse to believe that CCP does so.
|
Yuki Nagato
GoonFleet
GoonSwarm
|
Posted - 2008.02.20 04:51:00 -
[41]
Originally by: Herio Mortis
It's not sending 8 MB packets over the network
It actually is. Here, take a look at this thread.
|
Eleana Tomelac
Gallente
Through the Looking Glass
|
Posted - 2008.02.20 13:16:00 -
[42]
Originally by: Vocis
What does it matter if my target has his hardeners on?
All module activation with a visual effect needs to be sent to the other clients.
Originally by: Vocis
What does it matter if his guns are on?
Animation again.
Originally by: Vocis
The only thing that matters is if he is hitting me, and how much damage I am doing to him.
Then you can play a 2D, text or pen&paper strategy game.
Originally by: Vocis
To send completely worthless information to so many clients should be so stupid that I refuse to believe that CCP does so.
worthless if it sends anything that has no linked animation to it, otherwise, it's a 3D game, it's showing effects, no 'I want a text based strategy game' mode has been added yet, so it's sending effects to everyone.
--
Pocket drone carriers (tm) enthousiast !
Assault Frigates MK II |
Matrixcvd
Rionnag Alba
Triumvirate.
|
Posted - 2008.02.20 14:13:00 -
[43]
Wow, pretty interesting read, i am not a computer guy, but i would like to pose a question here. If the scalability of TQ does not improve, going back to heavy traffic advisories, and limiting the amount of people be the best solution?
|
Eleana Tomelac
Gallente
Through the Looking Glass
|
Posted - 2008.02.20 14:52:00 -
[44]
They are preparing a new cluster for sometime soon(tm)...
So, this should get better when it is out.
--
Pocket drone carriers (tm) enthousiast !
Assault Frigates MK II |
Yuki Nagato
GoonFleet
GoonSwarm
|
Posted - 2008.02.20 15:36:00 -
[45]
Originally by: Eleana Tomelac
They are preparing a new cluster for sometime soon(tm)...
So, this should get better when it is out.
If you're talking about sharding then no, they're not, and they've said "no" about a dozen times since I started playing.
As for Infiniband/RDMA I haven't heard anything about it. :(
|
Eleana Tomelac
Gallente
Through the Looking Glass
|
Posted - 2008.02.20 15:45:00 -
[46]
Read 'new' as 'replacement'.
I didn't say a thing about sharding.
They are always upgrading all the sol servers at the same time because it seems they need to be very similar for all to work smoothly.
--
Pocket drone carriers (tm) enthousiast !
Assault Frigates MK II |
Herio Mortis
54th Knights Templar
Dark Matter Coalition
|
Posted - 2008.02.20 21:09:00 -
[47]
Originally by: Yuki Nagato
Originally by: Herio Mortis
It's not sending 8 MB packets over the network
It actually is. Here, take a look at this thread.
It actually isn't, I've verified this myself. It does send large amounts of data at various occasions, but it plays nicely with whatever MTU I set. And yes, I am a network engineer by trade :)
@Eleana:
Yes, it does send updates on graphics etc. But this is not done between all the various clients as a mesh, nor is it meshed on the server. Doing that in a meshed way would just be silly as you say, the bandwidth requirements would be silly.
In the client the space scene and handling is in something called a ballpark. The manager process for this ballpark is as far as I can tell called michelle. My guess would be that just like godma it has a serverside replica. This would handle where people are in space in relation to each other, what visual effects are active etc.
This means that the ship handling threads on the SOL nodes do not have to mesh up and keep full track of all interactions themselves, this would be done in one process. I am also guessing that this ballpark manager is what we call a grid.
This means that the michelle stuff in the client only needs to get updated from its server counterpart instead of getting info from all other clients, thus severely diminishing the amount of processing needed for pilot interactions and visual effects.
Please note that all this is merely theorys and conjectures I've made from looking at the stuff in the EVE log viewer, some of it is just me pondering how I would have done it. I could be wrong, I could be right. It does make sense and feels logical to me atleast.
-----
And *pop* went my ferox... |
Allen Ramses
Caldari
Typo Corp
|
Posted - 2008.02.20 21:52:00 -
[48]
Maybe that means Michelle ate too many ballparks, and she's sluggish and cranky because of it.
...Sorry 
Or maybe, if it is the way you described it, one of these services is being overwhelmed?
And don't discount the exponential rise in bandwidth users with increasing active connections. I used to host a few UT99 servers (modded ones, TBE), and a single full 32 player server would require more than double the bandwidth of two full 16 player servers. I still believe that the nature behind this is the responsible culprit behind the lag.
____________________
Pimped out Raven to run level 4 missions quickly: 210 Mil ISK.
Realizing your 120 Mil ISK Drake gets the job done faster: Priceless. |
| |
|
| Pages: 1 [2] :: one page |
| First page | Previous page | Next page | Last page |