Pages: 1 [2] 3 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 15 post(s) |
lisaaa
|
Posted - 2011.06.20 17:48:00 -
[31]
Originally by: CCP Veritas
Quote:
CarbonIO has not been turned on for the client as yet, but since the client doesn't sling packets for a living that's not a big deal. It will need to be activated before BlueNet is leveraged at all, so you can expect it to happen before the multi-player Incarna release.
Quote:
CarbonIO has been live on TQ cluster-wide for a week.
I don't get >.<
|
Flynn Fetladral
Caldari BlackSite Prophecy
|
Posted - 2011.06.20 17:52:00 -
[32]
Great blog! Will be nice to see multi threaded client coming to a computer near you sometime in the future.
Follow Flynn on Twitter |
Callic Veratar
|
Posted - 2011.06.20 17:53:00 -
[33]
Originally by: Bagehi Graphs. So pretty. This leaves me wondering if someone, somewhere in the near future is going to rewrite the GIL so that it can send processes to other cores/CPUs to be run, rather than keep them all on the same core. But... I don't know much of anything about coding so, to whoever explains why this is a dumb idea, be nice please.
If I understood what was explained, the GIL is a transactional lock. Processing that require it's function must be done in the order they were received to prevent memory trampling. Think of it like this: I buy a ship, then I sell a stack of minerals. My bank balance should be [balance] = [balance] - [ship] + [minerals]. But, if you tried the transactions at the same time you could end up with [balance] = [balance] - [ship] with your minerals missing or [balance] = [balance] + minerals] with a free ship.
|
|
Fien Silver
|
Posted - 2011.06.20 17:53:00 -
[34]
Originally by: Miss Modus What was happening at the four peaks of the CPU% per user graph where the CarbonIO spiked much higher than the original StacklessIO?
Egads I was hoping that would slide by. I almost erased them out but thought it best not to fudge the data in any way.
These graphs are from the initial 24-hour deployment to a fully-loaded pair of proxy nodes on TQ. The "spikes" are the result of a spinlock-bug I fixed the next day. Essentially, since the communications is now multithreaded, its possible for one thread to call for the close of a socket at the same time another one is writing data to it. This condition is guarded against with a mutex of course, but if it occured at EXACTLY the same time (multi-core) there was a "hole" in the logic that spun a core until the socket was actually released. This had no effect on throughput, but would tie down a core until the TCP/IP stack finnaly came back and said "yup he's gone".
Spinlocks on mutli-core systems are more efficient in cases of small hold times and infrequent contention, but in this case I needed to use a more conventional sleep/event type lock.
Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)
Originally by: J Kunjeh Can't wait to see what this does to TQ in the wild.
Didn't I mention? You're soaking in it. Its been TQ-wide for the last week :) Was fun to watch our CPU usage drop off a cliff.
Originally by: Uncanny Valley Is BlueNet going to be available for the Incarna release, or just CarbonIO? When are you going to "publish" BlueNet data?
This was created to service the data requirements of Incarna, but in so doing, the door was opened for other systems to use it. CarbonIO was created so that something like BlueNet could function (ie- send/receive packets off the GIL). BlueNet is now available for internal teams to start examining and see if they can leverage.
Originally by: Ishina Fel Whoa, wait a moment! Where did that one come from? I heard no mention of this in any prior devblogs, and not in the fanfest coverage either Surprised And yet, this is the kind of thing people will frantically gobble up, the kind of thing everyone wants to read.
When are which parts of this getting enabled? Is that an official feature of the Incarna expansion, or just something that happened to be ready at the same time? What are you hoping to do with this technology in the future?
Well.. the simple version is we didn't know if this would work at all, let alone gain any efficiency. CarbonIO was written to open up a data-path through MachoNet that was wide enough to allow Incarna data to flow without hurting cluster performance, and we can show that it does that. Other than that, we didn't really expect any efficiency gains. But apparently the ground-up rework with this new paradigm *did* result in some gains.. some big ones.. but we only discovered that once we ran it, and then only over the last week and a half or so.
Something this big and basic being rolled into TQ, we never thought it would work the first time, that's why we took it so slow to begin with. But.. it DID work the first time, and fast, surprising everyone (me most of all).
So to sum up, we didn't tell anyone because we didn't know if we would have anything to tell, and even if we did, we didn't know when it would be available.
Set the way-back machine to 2 weeks ago with me huddled at my desk praying to the gods of no-crashy as this was rolled this into Jita for the first time. Hoping that whatever went wrong I'd be able to identify and fix before Incarna had to ship.
Yeah you didn't want to be me. "reproduction steps: get 2000 users to perform trade actions randomly on Jita.. ".
But I'll be damned if it didn't work, and lowered CPU by 10%.
|
|
|
CCP Curt
|
Posted - 2011.06.20 17:55:00 -
[35]
BAH. defaulted to my throwaway alt. That last post was meant to be by CCP Curt
|
|
lisaaa
|
Posted - 2011.06.20 18:00:00 -
[36]
What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???
|
Callic Veratar
|
Posted - 2011.06.20 18:04:00 -
[37]
Originally by: lisaaa
What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???
Exactly that, it's enabled on the server but not on the client yet. It doesn't need to be on both ends to work.
|
|
CCP Curt
|
Posted - 2011.06.20 18:07:00 -
[38]
Originally by: lisaaa
What did Veritas mean by that the CarbonIO isn't turned on for the client, but it has been live on TQ for a week now ???
He meant the TQ cluster, server-side. The client will see no benefit from this since it spends almost no time sending packets. We have not fully tested all the client systems with it yet, but will turn it on once we have. So far so good.
Originally by: Miss Modus What was happening at the four peaks of the CPU% per user graph where the CarbonIO spiked much higher than the original StacklessIO?
Egads I was hoping that would slide by. I almost erased them out but thought it best not to fudge the data in any way.
These graphs are from the initial 24-hour deployment to a fully-loaded pair of proxy nodes on TQ. The "spikes" are the result of a spinlock-bug I fixed the next day. Essentially, since the communications is now multithreaded, its possible for one thread to call for the close of a socket at the same time another one is writing data to it. This condition is guarded against with a mutex of course, but if it occured at EXACTLY the same time (multi-core) there was a "hole" in the logic that spun a core until the socket was actually released. This had no effect on throughput, but would tie down a core until the TCP/IP stack finnaly came back and said "yup he's gone".
Spinlocks on mutli-core systems are more efficient in cases of small hold times and infrequent contention, but in this case I needed to use a more conventional sleep/event type lock.
Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)
Originally by: Uncanny Valley Is BlueNet going to be available for the Incarna release, or just CarbonIO? When are you going to "publish" BlueNet data?
This was created to service the data requirements of Incarna, but in so doing, the door was opened for other systems to use it. CarbonIO was created so that something like BlueNet could function (ie- send/receive packets off the GIL). BlueNet is now available for internal teams to start examining and see if they can leverage.
Originally by: Ishina Fel Whoa, wait a moment! Where did that one come from? I heard no mention of this in any prior devblogs, and not in the fanfest coverage either Surprised And yet, this is the kind of thing people will frantically gobble up, the kind of thing everyone wants to read.
When are which parts of this getting enabled? Is that an official feature of the Incarna expansion, or just something that happened to be ready at the same time? What are you hoping to do with this technology in the future?
Well.. the simple version is we didn't know if this would work at all, let alone gain any efficiency. CarbonIO was written to open up a data-path through MachoNet that was wide enough to allow Incarna data to flow without hurting cluster performance, and we can show that it does that. Other than that, we didn't really expect any efficiency gains. But apparently the ground-up rework with this new paradigm *did* result in some gains.. some big ones.. but we only discovered that once we ran it, and then only over the last week and a half or so.
Something this big and basic being rolled into TQ, we never thought it would work the first time, that's why we took it so slow to begin with. But.. it DID work the first time, and fast, surprising everyone (me most of all).
So to sum up, we didn't tell anyone because we didn't know if we would have anything to tell, and even if we did, we didn't know when it would be available.
Set the way-back machine to 2 weeks ago with me huddled at my desk praying to the gods of no-crashy as this was rolled this into Jita for the first time. Hoping that whatever went wrong I'd be able to identify and fix before Incarna had to ship.
Yeah you didn't want to be me. "reproduction steps: get 2000 users to perform trade actions randomly on Jita.. ".
But I'll be damned if it didn't work, and lowered CPU by 10%
|
|
tatsudoshi I
Gallente The Venus Project - Zeitgeist Movement
|
Posted - 2011.06.20 18:07:00 -
[39]
Edited by: tatsudoshi I on 20/06/2011 18:09:15
Originally by: CCP Curt Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)
I actually thought that was what MissModus was asking about.. So what are the ever growing spikes and when or if do they stop? ..................................................
May we all have the courage to believe. Long live Mifune! |
|
CCP Curt
|
Posted - 2011.06.20 18:14:00 -
[40]
Originally by: tatsudoshi I Edited by: tatsudoshi I on 20/06/2011 18:09:15
Originally by: CCP Curt Now if you were really clever you'd ask about those little peaks that grow slowly over time toward the end of the run ;)
I actually thought that was what MissModus was asking about.. So what are the ever growing spikes and when or if do they stop?
They are stat crunching. The longer the server runs the more data it sifts through when polled for information. Toward the end of the run it actually starts to be visible. CarbonIO manages idle time quite a bit differently (not necessarily better, just different) than StacklessIO so while those spikes were previously lost in the noise, they became visible.
This is being addressed now to more efficiently deal with that data.
|
|
|
Motriek
Navy of Xoc Wildly Inappropriate.
|
Posted - 2011.06.20 18:16:00 -
[41]
Please clarify what portions of the server stack this impacts and benefits. As understand it, there are proxy servers, sol nodes / servers, and database servers. Does this primarily benefit the proxies or the sols?
|
|
CCP Curt
|
Posted - 2011.06.20 18:23:00 -
[42]
Originally by: Lykouleon CCP CURT, 010101110110100101101100011011000010000001111001011011110111010100100000011011010110000101110010011100100111100100100000011011010110010100111111
Seriously nice blog and some great info. And enough graphs to appease my graph-throne <3
cat nerd.cpp #include <stdio.h>
//------------------------------------------------------------------------------ int main( int argn, char *argv[] ) { char in[]="010101110110100101101100011011000010000001111001011011110111010" "100100000011011010110000101110010011100100111100100100000011011" "010110010100111111";
int pos = 0; while( in[pos] ) { char accumulator = 0; for( int i=0; i<4 && in[pos] ; i++ ) { accumulator += in[pos] == '1' ? 1<<i : 0; pos++; }
printf( "[%d:%c]:", (int)accumulator, accumulator ); }
printf( "\n" ); return 0; }
$ gcc -o nerd nerd.cpp ;./nerd [10: ]:[14:]:[6:]:[9: ]:[6:]:[3:]:[6:]:[3:]:[4:]:[0:]:[14:]:[9: ]:[6:]:[15:]:[14:]:[10: ]:[4:]:[0:]:[6:]:[11: ]:[6:]:[8]:[14:]:[4:]:[14:]:[4:]:[14:]:[9: ]:[4:]:[0:]:[6:]:[11: ]:[6:]:[10: ]:[12: ]:[15:]:
I don't get it.
-Curt
|
|
Sakura Ren Fenikkusu
|
Posted - 2011.06.20 18:29:00 -
[43]
Edited by: Sakura Ren Fenikkusu on 20/06/2011 18:29:01 I definetly love to read dev blogs and articles like this.
I'm a hobbyist/indie developer, working on a few small games, and trying to learn everything I can, so that in the future I can make something as great as Eve Online.
It is one thing to read articles about how things could be done, and how they have been done in the past, another to see what live MMOs are doing to make their games better from a technical standpoint.
Keep up the good work.
EDIT: CCP Kurt broke the forums....
|
|
CCP Warlock
|
Posted - 2011.06.20 18:30:00 -
[44]
Originally by: CCP Curt
Originally by: Lykouleon CCP CURT,
//------------------------------------------------------------------------------ int main( int argn, char *argv[] ) { char in[]="010101110110100101101100011011000010000001111001011011110111010" "100100000011011010110000101110010011100100111100100100000011011" "010110010100111111";
int pos = 0; while( in[pos] ) { char accumulator = 0; for( int i=0; i<4 && in[pos] ; i++ ) { accumulator += in[pos] == '1' ? 1<<i : 0; pos++; }
printf( "[%d:%c]:", (int)accumulator, accumulator ); }
printf( "\n" ); return 0; }
$ gcc -o nerd nerd.cpp ;./nerd [10: ]:[14:]:[6:]:[9: ]:[6:]:[3:]:[6:]:[3:]:[4:]:[0:]:[14:]:[9: ]:[6:]:[15:]:[14:]:[10: ]:[4:]:[0:]:[6:]:[11: ]:[6:]:[8]:[14:]:[4:]:[14:]:[4:]:[14:]:[9: ]:[4:]:[0:]:[6:]:[11: ]:[6:]:[10: ]:[12: ]:[15:]: [/code]
I don't get it.
-Curt
It says "Will you marry me?" - I think you forgot that its 8 bits per character...
|
|
Ambo
I've Got Nothing
|
Posted - 2011.06.20 18:30:00 -
[45]
Originally by: CCP Curt
I don't get it.
-Curt
http://www.roubaixinteractive.com/PlayGround/Binary_Conversion/Binary_To_Text.asp
Paste binary, click 'to text'
Also, awesome dev blog and AWESOME work. --------------------------------------
|
tatsudoshi I
Gallente The Venus Project - Zeitgeist Movement
|
Posted - 2011.06.20 18:33:00 -
[46]
Edited by: tatsudoshi I on 20/06/2011 18:36:38
Originally by: CCP Curt
I don't get it.
-Curt
Drink more or less coffee, which ever is the opposite of what you are doing at this very moment! (Read with over-exited speaker voice)
EDIT, Ambo love your toon! ..................................................
May we all have the courage to believe. Long live Mifune! |
Bienator II
|
Posted - 2011.06.20 18:39:00 -
[47]
thats a lot of effort to avoid touching the GIL. (Python was probably a bad longterm choice for the server logic but i guess you have good reasons why you can't migrate)
have you experimented with Jython (http://www.jython.org)? It is basically fully multithreaded python on the JVM but was between 0.1x and 5x slower (single threaded comparison) the last time i played with it. It is under active development and gets faster with every release.
So that might get interesting in future if you want to run on 32+ cores which isn't that uncommon for a java server any more :)
|
|
CCP Curt
|
Posted - 2011.06.20 18:45:00 -
[48]
Originally by: CCP Warlock
It says "Will you marry me?" - I think you forgot that its 8 bits per character...
<mirage>[curt|/home/curt]$ cat nerd.cpp #include <stdio.h>
//------------------------------------------------------------------------------ int main( int argn, char *argv[] ) { char in[]="010101110110100101101100011011000010000001111001011011110111010" "100100000011011010110000101110010011100100111100100100000011011" "010110010100111111";
int pos = 0; while( in[pos] ) { char accumulator = 0; for( int i=0; i<8 && in[pos] ; i++ ) { accumulator += in[pos] == '1' ? 1<<(7 - i) : 0; pos++; }
printf( "%c", accumulator ); }
printf( "\n" ); return 0; }
<mirage>[curt|/home/curt]$ gcc -o nerd nerd.cpp ;./nerd Will you marry me?
Ah no, sorry married with 3 kids already.
Coffee? heck no I go right to the source- Diet Coke.
|
|
Mike deVoid
Firebird Squadron Terra-Incognita
|
Posted - 2011.06.20 18:46:00 -
[49]
Always worth telling CCP employees and CCP teams when they do impressive work, to balance when I post in the rage threads.
GJ guys :). -----
Quote: The maximum acceptable limit of MT in EVE is vanity items + the ability to buy things already created legitimately by another player. PLEX can be already be used to attain SP, ships an |
|
CCP Curt
|
Posted - 2011.06.20 18:47:00 -
[50]
Originally by: Bienator II thats a lot of effort to avoid touching the GIL. (Python was probably a bad longterm choice for the server logic but i guess you have good reasons why you can't migrate)
have you experimented with Jython (http://www.jython.org)? It is basically fully multithreaded python on the JVM but was between 0.1x and 5x slower (single threaded comparison) the last time i played with it. It is under active development and gets faster with every release.
So that might get interesting in future if you want to run on 32+ cores which isn't that uncommon for a java server any more :)
I know that it gets examined from time to time along with other possible solutions, but that's way outside my paygrade. I just make the little box go *ping*
|
|
|
Korerin Mayul
Amarr
|
Posted - 2011.06.20 19:02:00 -
[51]
i like the smell of this. i hope this stuff gets some air-time with the 'high level languages in supercomputing' crowd.
|
Herr Nerdstrom
Caldari Havoc Violence and Chaos Merciless.
|
Posted - 2011.06.20 19:05:00 -
[52]
I have a couple questions:
- What is the percentage of operations that are affected by this offloading of GIL and non-GIL related code, and that therefore benefit from these changes? Optimizing code to reduce CPU usage of that code by 35% is great, but it's not that much of a benefit if that code only accounts for 3% of the load. - It sounds like a lot of the efforts here are basically to reinvent nonblocking I/O, and perhaps splitting the network I/O (including compression and encryption) off to another thread, with the inclusion of completion ports. What am I missing? - The final graph showed the difference between StacklessIO and CarbonIO CPU usage. However, the CPU being examined was never at 100% to begin with. Without testing these upgrades on a core that was maxed, i.e., during a fleet battle, how can we tell how much actual benefit will be produced? Providing stats of a core that was already functioning fine, and therefore without significant lag(TM), doesn't really tell us anything about the effect of the upgrades. If the changes will bring the CPU usage of a 1000 pilot fleet battle to something less than 100%, then I think CCP can say something about the effectiveness of the changes.
I still think this is CCP spending years trying to overcome the limitations of a 'convenient' language like Python. The better choice, in my opinion, still would have been to use a non-interpreted and fully-featured language like C or C++.
That being said, I appreciate the efforts of the various teams at CCP, and applaud their progress. This dev blog was also excellent...please keep them coming.
|
|
CCP Curt
|
Posted - 2011.06.20 19:27:00 -
[53]
Originally by: Herr Nerdstrom Edited by: Herr Nerdstrom on 20/06/2011 19:07:43 I have a couple questions:
- What is the percentage of operations that are affected by this offloading of GIL and non-GIL related code, and that therefore benefit from these changes? Optimizing code to reduce CPU usage of that code by 35% is great, but it's not that much of a benefit if that code only accounts for 3% of the load.
Sort of, but it can depend greatly on how you measure and define "load". I'm not going to p lay games, but it bears mentioning that if I define "load" as "time spent processing on a CPU" and that 3% is spent doing inefficient things that cause the other 97% to spin in circles, then clearly by eliminating it the overall efficiency gain can greatly surpass 3%.
I do think there was some of that going on, with the GIL being requested too often, the overall system efficiency went down, CarbonIO requests it less.
But lets take that point at face value- clearly it is not the case since the graphs show a very substantial gain, on the order of 50%, for proxies. So comm must have been taking up at least 50% of the load, yes? And most likely far more since reducing the entire comm library to a single NOP would probably not work.
Quote:
- It sounds like a lot of the efforts here are basically to reinvent nonblocking I/O, and perhaps splitting the network I/O (including compression and encryption) off to another thread, with the inclusion of completion ports. What am I missing?
Not re-invent.. redeploy.. StacklessIO was using completion ports and nonblocking I/O, but it was tightly coupled to the GIL. ie- there was built in seriality. On a whiteboard or flowchart it would be a simple matter of erasing some lines and penciling in "a miracle occurs" and *poof* decoupled, but in the real world it took a long development effort to remove.
The end effect is that not JUST communications occurs off-GIL (as StacklessIO did) but that the code USING it can ALSO occur off-GIL.
Quote:
- The final graph showed the difference between StacklessIO and CarbonIO CPU usage. However, the CPU being examined was never at 100% to begin with. Without testing these upgrades on a core that was maxed, i.e., during a fleet battle, how can we tell how much actual benefit will be produced?
You make a good point- if both graphs cross 100% at 1000 users, then their shape is irrelevant. If that shape conferred no information, but they do. Load growth follows fairly linearly at ever load we have tested live, and the labratory tests (where we pushed big-iron blades to 100% routinely) predicted savings.
So far the lab data has been borne out on live loads, which can be taken as evidence that other lab predicitons are likely to be true. We can't be sure until its actually tried.. and we are standing by.. but a reasonable conclusion based on all available data is that there will be some savings on near-100% loads.
Which is irrelevant!
CarbonIO was never about reducing load directly, that's a side effect! BlueNet is about reducing serial load, when teams like Gridlock start using it to go-wide on the flying-in-space systems.
|
|
tatsudoshi I
Gallente The Venus Project - Zeitgeist Movement
|
Posted - 2011.06.20 19:35:00 -
[54]
Edited by: tatsudoshi I on 20/06/2011 19:35:37
Originally by: CCP Curt Coffee? heck no I go right to the source- Diet Coke.
I have heard around the health-heads that diet drinks on long term stores "something" in the bone marrow or something. Because there is no energy in the drink the body does not "see" the drink as something that needs to be digested so it does not know what to do and so it stores it.. or something. I have not researched this for health-fanaticism so use as seen fit. Only FYI. I have switched to coffee as I can't drink liters of it as I could with soda. ..................................................
May we all have the courage to believe. Long live Mifune! |
pmchem
Minmatar GoonWaffe Goonswarm Federation
|
Posted - 2011.06.20 19:35:00 -
[55]
CarbonIO sounds great (nice work). If I may redirect a little bit, any chance you guys will be able to use a MPI implementation for python (or... any underlying server code) anytime soon for taking _real_ advantage of having multicore CPUs?
This would be the ultimate solution for server-side scaling of fleet fights, as you're probably already aware.
|
Chaotic Alacrity
|
Posted - 2011.06.20 19:50:00 -
[56]
Seriously, half the reason I play this game is so that I have context for the awesome devblogs. Great job.
Are the internal discussions regarding new optimizations made possible by CarbonIO centered on reorganizing the calls to existing modules or are people starting to look for ways that they could offload logic currently in python to external libraries (that can now run concurrently)?
|
Klandi
Science and Trade Institute
|
Posted - 2011.06.20 20:32:00 -
[57]
Great post - lots of detail and juicy bits
Looking forward to lots more - as much or as deep as you want to go
|
Sri Nova
|
Posted - 2011.06.20 20:38:00 -
[58]
im surprised the developers of python have not come over there, to give all of you a good beating. For abusing their environment in ways that they could never have imagined.
Its amazing what ccp has pulled off with this engine and its fascinating reading about your accomplishments . im sure a lot of head banging, hair pulling, and sending off dear friends to the loony bin occurred during all this .
I bet all of you are elated when you make gains like this .and no words can adequately commend you guys for the hard work you put in to this .
but awesome job guys really its appreciated especially when we are playing the game, enjoying it and not realizing that its your hard work enabling us immerse ourselves in your universe so easily .
|
Glyken Touchon
Gallente Independent Alchemists
|
Posted - 2011.06.20 20:41:00 -
[59]
Great job!
Originally by: CCP Curt Egads I was hoping that would slide by. I almost erased them out but thought it best not to fudge the data in any way.
These graphs are from the initial 24-hour deployment to a fully-loaded pair of proxy nodes on TQ. The "spikes" are the result of a spinlock-bug I fixed the next day.
Good choice. Real data with an explanation comes across much better with this audience. ______
When the forums asked CCP for transparency, we didn't mean the HUD... |
Salene Gralois
K-2
|
Posted - 2011.06.20 21:29:00 -
[60]
That'll cure me of my geek-itch for a week. Thanks, i loved this type of blog in the past and (surprise :D) i still love them.
|
|
|
|
|
Pages: 1 [2] 3 4 :: one page |
First page | Previous page | Next page | Last page |