Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Xereyn
|
Posted - 2010.07.28 06:03:00 -
[1]
Disclaimer: I have never, personally, programmed an online game involving tens of thousands of active connections at a time.
That said:
Okay, everyone agrees that the lag problem sucks. It's clear that CCP are having trouble tracking it down. At this point, it seems possible to me that they may be dealing with one of those problems that turn out to be face-palmingly obvious when actually located, despite the fact that the coder(s) spent weeks/months/whatever looking for them.
Now, a common way for most of the programmers I know to solve these problems is to complain about it to friends/family members/the cat, and either have a blinding moment of inspiration in the process of explaining it, or else have the explainee ask an ignorant and foolish question that turns out to be critical.
So, for the hell of it:
What do you think might be causing the lag problem?
Generalised complaining not encouraged. Creativity encouraged.
Mine:
The fleet finder. Just because it would be hard to test properly on Sisi, since you don't have the general server population/other random gangs hanging around confusing things, and it's something that affects fleet actions, and it's potentially a sneaky little rat that slips under the radar but does weird things to processing.
Anyone else have ideas? Come on! Instead of just whining, let's see if we can provoke some sort of moment of blinding insight in some developer somewhere that lets them fix the lag at last.
|
Barkaial Starfinder
Minmatar The Kairos Syndicate
|
Posted - 2010.07.28 06:13:00 -
[2]
It's not a new feature, but I always thought drones consumed a lot of resources from the servers. They each seem to act like individual ships.
|
Kireiina
|
Posted - 2010.07.28 06:19:00 -
[3]
They don't really know what their code is doing (since they can't track a regression in 9 months of trying) so it is fairly pointless for us to try. However the fact that they are so stumped by the situation makes it pretty clear their code is a terrible mess and / or they've lost a lot of design knowledge.
Virtually no software provider would get away with having a serious introduced fault still active with no time-line for when a fix may occur.
I'd suspect that maybe building the server out of python wasn't a very clever idea. I like the language but it would not be my first choice for producing structured code and trying to do performance debugging on something that is running in an abstracted environment (whether interpreted or bytecode VM) could so easily get painful.
|
Emperor Cheney
Celebrity Sex Tape
|
Posted - 2010.07.28 06:23:00 -
[4]
How's this for a credible conspiracy-ish theory:
They know exactly what caused it - the features of dominion such as fleet finder. However, orders from management are that no rollback will happen. So the designers assigned are really trying to trim out other things to make it work as best as possible, considering.
|
Phantom Slave
Universal Pest Exterminators
|
Posted - 2010.07.28 06:23:00 -
[5]
- Nodes could be attempting to send too much information to each other when players change systems. When you have a ton of people jumping at once it could be the servers are sending too much data/repetitive data back and forth.
- Somebody forgot to turn off the /test script in the code somewhere so the servers are processing all their data through an additional function somewhere and is causing lag.
- Drone AI has changed quite a few times, and it could be a new change that's causing too much CPU usage.
- An Algorithm has changed and wasn't noticed because somebody was just testing something and forgot to change it back, causing the server to use non-optimized math for its side of the simulation.
- Smack talk has hit an all-time high and is causing lag because of the amount of data being sent.
- Server could be sending dummy information to a dummy client or unknown clients (non existent).
- Server processing order has changed, it no longer focuses most of its resources on updating the new people in local and is spending more time on other things.
I'm tired, can't think of much else and there's probably some repeats in there somewhere.
|
mechtech
SRS Industries SRS.
|
Posted - 2010.07.28 06:34:00 -
[6]
Fleet finder?
Something got much worse in Dominion.
|
Aion Amarra
Minmatar Real Nice And Laidback Corporation Black Core Alliance
|
Posted - 2010.07.28 08:23:00 -
[7]
Watchlists?
|
Zartrader
|
Posted - 2010.07.28 08:34:00 -
[8]
They accidentally introduced the forum 5 minute posting delay to fleets or their code is bloated, badly documented and past a critical point of no return.
|
Natalie Caladan
|
Posted - 2010.07.28 08:39:00 -
[9]
How much people are actually participating in fleet battles? I thought the vast majority of players were highsec mining/missioning carebears?
|
Miss Connolly
Public Relations Corp
|
Posted - 2010.07.28 09:07:00 -
[10]
We have to differentiate between normal "lag" and the "lag" we are currently experiencing. EVE always lagged (even at the best of times before the release of Dominion) but it remained well within playable limits. Modules would take a while to activate and de-activate, grid loading might take a little longer then the "intended" 30 seconds (I'm assuming this is the maximum duration that CCP expected a grid to load - that's why you remain invisible for 30 seconds). But all in all it remained very playable even if it did "lag" somewhat.
What we are experiencing now on the other hand is much more then simple lag. At some point the node just starts dying and then things just go from bad to worse - guns start locking up, shooting/MWDing without using ammo or cap, launching drones takes 2-3 minutes, grid loading starts taking unacceptably long (10-15 minutes) and thr overview starts showing non-existent targets etc.
However this is not actually "lag" in the sense that the latency between the server and clients is too high - the problem is server-side performance. The client sends information to the server (if this step takes too long then it's classic "lag"), the server receives the information and then sends it back to the client (again, this step could potentially cause lag). However in the case of EVE it's not the sending and receiving that is causing the problem - that might account for a few hundred milliseconds but not for a 15 minute grid load. The problem is basically the mathematical complexity of calculating all activity on a grid with 1000 people on it. Just imagine the amount of data that the node has to handle: every ship has 7 or 8 guns (which we have to ungroup during high lag situtations - thus further increasing the load on the server) and 5 drones. Every ship and drone has a movement-vector and every gun and drone that fires has to have the server calculate range to target, tracking, current signature of the target and the resulting damage reduction etc.etc.pp. What further amplifies the problem is the fact that the server load doesn't increase in a linear fashion - to calculate a grid with 10 people takes X CPU time but to calculate a grid with 100 people doesn't take X * 10. The increase is more like X^10 because the server has to tell each client the position of each ship on the grid - so for 2 ships that's 4 positions and vectors the server has to provide to the clients. For 3 ships it's already 9 positions and vectors that have to be communicated and for 100 ships it's already 10'000 vectors and positions that the node has to coordinate and provide to the clients in a timely fashion.
This is not a problem that can be solved - ever. It's in the nature of MMOs that they have scaling issues. Why do you think all the other MMOs use instances and sharding? The reason is to keep the above mentioned complexity at a manageable level.
The only solution is to re-design the game so numbers are no longer an advantage - or by forcing large groups to split up into multiple battlefields who's complexity remains manageable.
CCP had this opportunity only a few months ago during the Sov re-design. Sadly they failed to make use of the opportunity so now we're stuck.
However, this doesn't change anything about the fact that the game did run much better before the release of Dominion. So either something in Dominion added to the complexity of calculating a grid with many people or there is some memory leak or bug in the server code that is causing the co-ordination process on the node to start failing.
I know it's naive to simply say "compare the two releases and see what changed" but basically that's what it comes down to. Setup a second test-server called SiSi2 with pre-dominion code, offer players 500k SP or a free day of game time if they take part in the mass testing and then have two fleet fights: first one on SiSi2 running the old code and then a second fight on SiSi with dominion code (with equal numbers).
|
|
Spacing Cowboy
Caldari Rule of Five Lucky Starbase Syndicate
|
Posted - 2010.07.28 09:23:00 -
[11]
Options brainfarted:
Wormhole's , the entery's & exits to complex to handle ? Sleeper AI , there a tad bit smarter then a regular npc Keeping up with indexes in SOV space ?
Market.. did the market ( buy / trade ect ) became more active and overloading the old system ?
The 2GB "a pop" downloads of new players starting to deep-fry the servers, also effecting the game ?
|
Ryhss
Caldari Ominous Corp Circle-Of-Two
|
Posted - 2010.07.28 09:54:00 -
[12]
Originally by: Kireiina
Virtually no software provider would get away with having a serious introduced fault still active with no time-line for when a fix may occur.
There's a time line. In 18 months they'll start looking at the problem.
Originally by: Abrazzar I don't know what's going on but I am terribly upset about it.
|
Susan Fiona
Independent Coalition
|
Posted - 2010.07.28 10:44:00 -
[13]
Rewrite the whole code.
Yes I know this takes a while and takes a lot of resources, but given the amount of time and effort you can spend trying to fix the current code, why not redesign it, start "all over" ?
Maybe even change language of which it's written in.
As said I know this will take a while, but how long would it take to chase down a bug in the current code?
/Susan
|
Amerilia
|
Posted - 2010.07.28 10:53:00 -
[14]
Originally by: Susan Fiona Rewrite the whole code.
Yes I know this takes a while and takes a lot of resources, but given the amount of time and effort you can spend trying to fix the current code, why not redesign it, start "all over" ?
Maybe even change language of which it's written in.
As said I know this will take a while, but how long would it take to chase down a bug in the current code?
/Susan
Rewrite? Currently they have 11 people working on the core game, if those were to try to redo everything, I¦d say 10 years or more. ( a while? Are you kidding me?) Finding the bug? Well depends on how lucky you are, you can find a bug immediately or chase it for years, of course there are a lot of factors that can affect it.
|
Susan Fiona
Independent Coalition
|
Posted - 2010.07.28 11:02:00 -
[15]
Originally by: Amerilia
Originally by: Susan Fiona Rewrite the whole code.
Yes I know this takes a while and takes a lot of resources, but given the amount of time and effort you can spend trying to fix the current code, why not redesign it, start "all over" ?
Maybe even change language of which it's written in.
As said I know this will take a while, but how long would it take to chase down a bug in the current code?
/Susan
Rewrite? Currently they have 11 people working on the core game, if those were to try to redo everything, I¦d say 10 years or more. ( a while? Are you kidding me?) Finding the bug? Well depends on how lucky you are, you can find a bug immediately or chase it for years, of course there are a lot of factors that can affect it.
Part of the whole "Taking a lot of recourses" I mean they should hire some more programmers, obviously, and get it done faster.
|
Hemp Invader
EVERYTHING IS TERRIBLE Prime Orbital Systems
|
Posted - 2010.07.28 11:18:00 -
[16]
my guess is:
-fleet finder(since everybody is in a fleet when lag happens and there have been some issues with it in the past). My thought was, remove fleet finder code and redesign it from scratch.
-drones. If i were in CCP's shoes i'd do drone ai stacking.(only one drone controls the rest) and if that doesn't help, i'd remove drones and boost gun damage and that's it tbh.
-gun stacking doesn't get used by players so much anymore due to module lag. Encourage the use of gun stacking by all means possible.
Just my 2 isk cents.
|
Bitter Veteran
Brutor tribe
|
Posted - 2010.07.28 11:19:00 -
[17]
Station RF timers.
Every time i lag out and black screen their is usually on in system.
|
Hari Markkus
Freelancing Corp Confederation of Independent Corporations
|
Posted - 2010.07.28 11:25:00 -
[18]
Big fleet fights have always lagged. The problem just now is that before the last expansion the size of fleet that caused lag was larger than now.
So either each member of the fleet is taking up more resources or the system in general is busier before the fleet action takes place and so there is less resources to go around.
My guess is that it is CPU time that is the limiting factor, but it could be any resource. The only way to find out for sure is to instrument the code and do some measurements (but of course instrumenting the code makes it slower leading to more lag ).
If lack of CPU is the problem then a rewrite of critical sections, getting a better complier, discarding some functionality and hardware upgrades will all help.
|
Mr Kidd
|
Posted - 2010.07.28 11:41:00 -
[19]
Quote: They don't really know what their code is doing (since they can't track a regression in 9 months of trying) so it is fairly pointless for us to try. However the fact that they are so stumped by the situation makes it pretty clear their code is a terrible mess and / or they've lost a lot of design knowledge. They know exactly what caused it - the features of dominion such as fleet finder. However, orders from management are that no rollback will happen. So the designers assigned are really trying to trim out other things to make it work as best as possible, considering.
That would generally indicate a cessation of active development and commencement of managing the game. And what I suspect is going on with expansions is merely re-utilizing existing features with minor updates of those existing features in order to produce new(TM) features.
Quote:
They know exactly what caused it - the features of dominion such as fleet finder. However, orders from management are that no rollback will happen. So the designers assigned are really trying to trim out other things to make it work as best as possible, considering.
That's not "exactly" anything. You generalized. It's very probable that a newly enabled feature would conflict with much older code. Or cause older code to be executed more than it needs to. Unless one were to re-theorize the logic and compare it to every line of code, it's going to be impossible to fix. I would guess that the lag after Dominion is due to some processes that are being called upon more than they should as compared to Apocryphia. However, it doesn't really help knowing that if the current game managers have no idea what those processes could be or have any means to monitor them without going back through the code line for line.
What the current problems do indicate is a copy, paste and tweak mentality on the part of the game managers in order to introduce new(TM) features.
|
Average Jack
|
Posted - 2010.07.28 11:45:00 -
[20]
Originally by: Hari Markkus The only way to find out for sure is to instrument the code and do some measurements (but of course instrumenting the code makes it slower leading to more lag ).
It can't get much worse then now. The game is already unplayable so if the grid takes 10, 15 or 30 mins to load really doesn't make much difference.
They simply need to communicate it beforehand (via a technician that can explain things properly and not some marketing douche). Nobody will be angrier then they already are now (for example I'm an inch away from quitting and I'd rather suffer through a bit more lag for a few weeks then a little less lag for 18 months).
|
|
Jita Dancer
|
Posted - 2010.07.28 11:48:00 -
[21]
Last night, I was in C-J with my main. We had 8-900 in local. 75% blue. Blues were whacking SBU's, Reds were whacking a POS. We didn't meet head to head, but we both had massive fleets, engaging fixed structures. Lag was non existent - well perhaps Jita levels at worst. If someone jumped through the gate while we were on it shooting SBUs then they loaded quick enough to start burning back to the gate - although with 500 reds shooting them they never made it - point being they were loading FINE jumping into system in isolation.
The previous night we had 1050 in local 50/50 red/blue trying to slug it out with each other - total treacle. Module lag was minutes. In 2 hour fight I shot 30 rounds of ammo. (yawn). Both sides were shooting primaries (i.e. only a few ships were taking damage at a time) Admittedly there were more 'active targets' than the fixed targets of last night, but it wasn't like 500 people shooting 500 other people. More like 5 groups of 200 people all shooting 1 target each.
Node was reinforced on both occasions AFAIK.
Make of that what you will, but in my opinion, its **something** to do with the ship-to-ship interaction with opposing ships ON GRID i.e. damage calculations, gang bonuses, remote effects (repping/boosting/ecm etc) - its NOT just a factor of people in system. Well not always anyhow :) The only other differing factor was that on the first night there were 5+ titans and 30+ superacarriers (all red unfortunately :) that were primarying/instapopping logistics etc every ten minutes. Maybe Fighter bombers and/or the new DD are worth a closer look.
The real problem is that both nights were boring beyond belief. A massive fight, in slow motion and shooting 100million hit-points of SBUs. Someone remind me why I log in again?
SK
|
Makko Gray
Gallente Nexus Aerospace Corporation The Volition Cult
|
Posted - 2010.07.28 12:08:00 -
[22]
Excess of data to the clients perhaps? We know about setting overview and brackets to help client performance but does the data for this all come down anyway, i.e. every celestial object, every drone. That'd probably be more client side/connectionr elated lag.
As for server lag, um:
- collision culling and bounding box calculations
- not using rule of sarrus in determinant calculations
- intersection calculations (probably not so much as ammo hits whatever regardless of obstructions)
- calculating attributes affected by various mods such as armor resists affected by ships skills and modules when being attacked by multiple targets
- using bubble sort
I imagine all the graphics stuff like shading, culling, rendering is all on the client and the server is just reduced to matrices and vectors, and damage calcultions or any of the other miriad of things that have to be calculated.
Is market stuff handled by seperate servers? Is there some sort of api they use to register your location and determine what market to show you?
Hard to know really without having more of an overview of the architecture, and there are several types of lags as well main ones obviously being server, client and connection lag (or bandwidth issues).
Then there's things like threading which you have to consider, any threads causing undesirable locks that sort of thing.
|
Per Bastet
Amarr B.O.O.M Obsidian Mining Coalition
|
Posted - 2010.07.28 12:28:00 -
[23]
It might have something to do with people and Sub Standard Internet Connections.
If your Phat Pipes are not so phat you slow everyone else down because in order for everything to work, you have to know where every one is, this means sending and receiving positional Information for everyone on Grid. More on Grid, More data that needs to move. If some one is on a low end connection it's going to induce lag, the more with low end connections, more lag.
Having read all of CCPs Tech Posts I don't think that it will be so much on their end, I think it's going to be outside their control to a great extent. --
Bastet's Organization Of Mining |
codemaster28
Caldari Phobos Alliance
|
Posted - 2010.07.28 12:40:00 -
[24]
Edited by: codemaster28 on 28/07/2010 12:41:54 Everyone who has played eve for the past 5+ years knows about lag, in past days ccp have reduced the amount of "objects in space" to reduce lag. Which to some degree has worked. Some prime examples of this was the drone nerf ie less drones in space at any given time. The problem is that ccp still havent optimized this throughout eve.
Examples of un-needed objects in space causing more load on servers Collidable objects ( ever warped into a mission to find 100 of the same object ), these could be made into a few objects Various gas gloud graphics etc
I agree with most people that 500 vs 500 man fleet on a non-reinforced node is not possible to have a decent fight , they should do what they did with missions - instead of trying to load everyone at the same time it should load in steps like missions do with npc`s load 50 wait 10 seconds load 50 wait 10 seconds load 50 etc etc , if your the 250th player to jump on that grid you should not appear on grid until you are fully loaded in your spawn group, maybe make it so you dont decloak untill your fully loaded "extended cloak". Just an idea.
|
FreddyMac
|
Posted - 2010.07.29 02:13:00 -
[25]
Originally by: Per Bastet It might have something to do with people and Sub Standard Internet Connections.
If your Phat Pipes are not so phat you slow everyone else down because in order for everything to work, you have to know where every one is, this means sending and receiving positional Information for everyone on Grid. More on Grid, More data that needs to move. If some one is on a low end connection it's going to induce lag, the more with low end connections, more lag.
We arent playing an FPS on a LAN setup here. Others in an mmo, dont lag because you have a crap connection.
Having read all of CCPs Tech Posts I don't think that it will be so much on their end, I think it's going to be outside their control to a great extent.
|
Apollo Gabriel
Domini Lex Talionis Etherium Cartel
|
Posted - 2010.07.29 02:51:00 -
[26]
I think they changed they way the environment is drawn and are overdoing it. Less is more.
I lag often alone out in space, on a T1.
Best, Apollo CCP Commit to Excellence by September 1st 2010
Don't let the trolls, keep you from your goals. |
kano donn
|
Posted - 2010.07.29 03:12:00 -
[27]
I believe the issues is from - The overview - Fleets - Standings
These three things were modified and since lag has gotten worse
What i notice: - Lag is generated when on the field when you have a lot of overviews trying to distinguish between friend and foe.
That is likely why turning overviews and brackets off is so helpful.
this issue is causing all the problems we see later. - Mod lag - Drones being silly - lag in other systems
|
Cipher Jones
Minmatar
|
Posted - 2010.07.29 03:57:00 -
[28]
Read the dev blogs and you will understand the situation a lot better. Been over this on this forum several times including as recently as today.
Really, the dev blogs explain a lot. At first I felt like a total fanboi, then I STFU and learneded. This is clearly a signature. |
Obsidian Hawk
RONA Legion
|
Posted - 2010.07.29 04:18:00 -
[29]
Here Read this before offering Ideas.
|
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |