Pages: 1 2 [3] 4 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 20 post(s) |
Puer Servus
Republic Military School Minmatar Republic
3
|
Posted - 2015.10.14 05:50:25 -
[61] - Quote
Cor'len wrote:Bienator II wrote: Thats prob why ccp seems to see MT as low priority atm. Actually, CCP would love to multithread the ~space code~ (can't remember the component name, haha). But it's practically impossible to get a consistent result; operations must be done in sequence, otherwise you get dead ships killing living ships, and other ~exciting~ edge cases. This is the ultimate limiter on EVE performance. They might conceivably be able to MT the processing of different grids in a single system, but everything that happens on a single grid must execute in a deterministic fashion, and in the correct order. Plus, even if that wasn't a problem, they run Stackless Python, with the beloved global interpreter lock which effectively prevents multithreading. tl;dr CCP wants to multithread all the things, but it's so hard it's bordering on impossible. Hence the effort to not have big fights.
Actually there is a ~space code~ multithread project called The Destiny Dispatcher.
http://www.youtube.com/watch?v=NEJbwZCgNgU&t=1h16m00s http://www.youtube.com/watch?v=UcEUB6h4Br0&t=8m10s
Can devs give any update on this project? |
Corraidhin Farsaidh
Farsaidh's Freeborn
1738
|
Posted - 2015.10.14 07:58:45 -
[62] - Quote
The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D
Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks... |
|
CCP DeNormalized
C C P C C P Alliance
304
|
Posted - 2015.10.14 09:43:49 -
[63] - Quote
Gospadin wrote:xrev wrote:Gospadin wrote:I'm shocked that a system designed to deploy in 2016 is even using rotating drives. That data must be REALLY cold. It's called auto-tiering. The hot storage blocks reside on the fast SSD's or the internal read cache. When blocks of data aren't touched, they move to slower disks that are still more cost effective if you look to volume for your buck. Compared to Ssd's, hard disks suck at random i/o but serial streams will do just fine. I know how it works. It's just interesting to me that TQ's cold data store is satisfied with about 10K IOPS across those disk arrays. (Assuming 200/disk for 10K SAS and about 50% utilization given their expected multipath setup and/or redundancy/parity overhead)
the DB averages around 2K IOPS during a regular run and while we spike upwards of 60-70K IOPS during startup, typically things are somewhat calm (2,000 batches per second @ the DB layer isn't massive by any means, but it's also far from quiet)
So in the end we have a flash tier of 5+ TB with a DB that's only 3 TB, plus we have over 700 GB of RAM for buffer pool space.
We really just don't need anything faster :)
CCP DeNormalized - Database Administrator
|
|
|
CCP Gun Show
C C P C C P Alliance
13
|
Posted - 2015.10.14 09:57:35 -
[64] - Quote
xrev wrote:San storage enthousiast here :)
Keep in mind before upgrading from 4 or 8 Gbps to 16 Gbps to check the available and used buffer credits in combination with the average transmitted package size. Default FC frame size is supposed to be 2112 bytes and uses 1 buffercredit to transmit. With the increased speed, more buffercredits are used to fully utilize the line. When they're used, nothing will be transmitted until buffers are freed up. Had a customer some time ago who had upgraded the linespeed and in combination with synchronous storage replication, production came to a halt because of the switches running out of buffers.
So CCP, please double check the framesizes and buffercredits before upgrading the speed ;)
Oh, just reminded, Iirc, the 48B-5 switches have 2 Condor3 asics, dividing the 48 ports in 0-23 on Asic1, and 24-47 on asic2. Keep in mind there's a default oversubscription on the asics of 1.5:1. So make sure to distribute the FC ports of a host evenly over both asics to control the performance :)
Cool stuff though. Make pictures when it arrives in the datacenters!
Excellent advice , Thanks for the constructive feedback |
|
|
CCP DeNormalized
C C P C C P Alliance
304
|
Posted - 2015.10.14 10:00:05 -
[65] - Quote
Corraidhin Farsaidh wrote:The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D
Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks...
Oracle RAC is super sexy for sure! But as you say, the licensing is nuts and at this point we just don't see any need to switch.
Cost / benefit just isn't there, and really MS SQL is quite reliable and has great HA/DR features with AlwaysOn and Availablity Groups.
we'll be doing some fun tests where we have a 4 node cluster with multiple AlwaysOn read secondary's: 2 nodes in our primary data center with a 3rd node in our DR Site and finally a 4th Node hosted @ amazon.
This is level of DR/BC that we're happy with :)
CCP DeNormalized - Database Administrator
|
|
virm pasuul
FRISKY BUSINESS. No Handlebars.
319
|
Posted - 2015.10.14 10:23:46 -
[66] - Quote
I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve? I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.
|
Xian Reevs
Armilies Corporation
0
|
Posted - 2015.10.14 10:38:19 -
[67] - Quote
I love when they talk dirty like that. Geek porn.
I am just wondering if anyone have even considered Hyper-V clustering? And no, I don't want to start a Hyper-V vs SEXi discussion. ;)
CCP DeNormalized wrote:We really just don't need anything faster :)
There is no such thing as "fast enough" for an IT enthusiast! |
Corraidhin Farsaidh
Farsaidh's Freeborn
1739
|
Posted - 2015.10.14 10:53:16 -
[68] - Quote
CCP DeNormalized wrote:Corraidhin Farsaidh wrote:The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D
Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks... Oracle RAC is super sexy for sure! But as you say, the licensing is nuts and at this point we just don't see any need to switch. Cost / benefit just isn't there, and really MS SQL is quite reliable and has great HA/DR features with AlwaysOn and Availablity Groups. we'll be doing some fun tests where we have a 4 node cluster with multiple AlwaysOn read secondary's: 2 nodes in our primary data center with a 3rd node in our DR Site and finally a 4th Node hosted @ amazon. This is level of DR/BC that we're happy with :)
What's funny is you most likely have better DR availability and testing than most of the major banks I've worked at. I wish they would pay half the attention to detail as you have :D
Ed: Actually I don't. I'd be out of a job then.... |
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
5626
|
Posted - 2015.10.14 11:23:41 -
[69] - Quote
Oracle licensing. ewww.
1 CPU license required per 2 cores. (for most intel kit) at around 32K per license. Then there's 15k for each RAC license (1 per 2 cores). And then there can be all the addins, which can double that easily. And to become liable for licensing, it can be as simple as 'run one query' or 'change one configuration setting')
Yes, I'm bitter
Woo! CSM X!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Nafensoriel
KarmaFleet Goonswarm Federation
124
|
Posted - 2015.10.14 12:05:21 -
[70] - Quote
virm pasuul wrote:I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve? I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.
You would be amazed at how many physicists, geneticists, chemists, etc play EVE... Not even considering the "just graduated line members" we have access to a veritable horde of skilled technical and amazing minds.
It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline. |
|
Natya Mebelle
Center for Advanced Studies Gallente Federation
314
|
Posted - 2015.10.14 12:41:33 -
[71] - Quote
Every now and then I read an Eve tech article to see how much I understand. It sounds foreign and astounding as usual, because I really have no point of reference even with the statistics for what will be done and all. But for all that tech talk, it leaves me with little that I can actually take away from the devblog to look forward to :c Even browsing the 4 pages on the forum didn't get me anywhere further.
So it leaves me with two big questions:
One... because I'm curious! How much does the entire thing cost in total?
Two... what will ACTUALLY change for us players?
Will session timers will be further reduced? Will transition between systems be faster? Will we get less dreaded "Socket closed" or other connectivity error messages? Because I know not every single one of them is related to the user. Will there be less random Tidi now in remote systems that are hardly populated just because the performance is needed elsewhere? And most importantly... are there plans with all that new power under the hood to fight the war against the 1second server tick? c:
oh and one that I forgot: With all the "redundancy" in the best way, is there a chance to look at no downtime or even faster downtime than we have nowadays? |
Alundil
Isogen 5
1040
|
Posted - 2015.10.14 13:12:10 -
[72] - Quote
Bienator II wrote:having DT only every second day would be a start :P Congratulations. You've succeeded in nerfing capital escalations by 50%
I'm right behind you
|
Corraidhin Farsaidh
Farsaidh's Freeborn
1741
|
Posted - 2015.10.14 13:16:27 -
[73] - Quote
Nafensoriel wrote:virm pasuul wrote:I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve? I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.
You would be amazed at how many physicists, geneticists, chemists, etc play EVE... Not even considering the "just graduated line members" we have access to a veritable horde of skilled technical and amazing minds. It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline.
One of the reasons why I like the game :) |
Corraidhin Farsaidh
Farsaidh's Freeborn
1742
|
Posted - 2015.10.14 13:27:44 -
[74] - Quote
Steve Ronuken wrote:Oracle licensing. ewww.
1 CPU license required per 2 cores. (for most intel kit) at around 32K per license. Then there's 15k for each RAC license (1 per 2 cores). And then there can be all the addins, which can double that easily. And to become liable for licensing, it can be as simple as 'run one query' or 'change one configuration setting')
Yes, I'm bitter
Addins which include the Grid Control stuff which is pretty much a necessity! I too am not at all bitter about it. Mainly because I never have to pay for it, my clients do :D |
Tiddle Jr
Brutor Tribe Minmatar Republic
592
|
Posted - 2015.10.14 13:36:31 -
[75] - Quote
i'm excited! And you? |
Esrevid Nekkeg
Justified and Ancient
513
|
Posted - 2015.10.14 14:02:23 -
[76] - Quote
Nafensoriel wrote:... It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline. Ships Carpenter here (woodworking on pricey yachts in my case)...Natya Mebelle wrote:Every now and then I read an Eve tech article to see how much I understand. It sounds foreign and astounding as usual, because I really have no point of reference even with the statistics for what will be done and all. But for all that tech talk, it leaves me with little that I can actually take away from the devblog to look forward to :c Even browsing the 4 pages on the forum didn't get me anywhere further. ... Same here. But I am more than willing to relax in the back seat of this limo enjoying the ride wile the technicians discuss the things going on under the hood.
Thanks CCP for the hard and undoubtedly necessary hard work on keeping TQ reliably going, now and in the future!
Here I used to have a sig of our old Camper in space. Now it is disregarded as being the wrong format.
Looking out the window I see one thing: Nothing wrong with the format of our Camper! Silly CCP......
|
Aryth
GoonWaffe Goonswarm Federation
1866
|
Posted - 2015.10.14 14:51:45 -
[77] - Quote
In the writeup I don't see why IBM. Did you bake these off against UCS and they were faster? Or just whitebox.
Maybe your performance needs are very niche but I have yet to see a bakeoff where IBM won against almost anyone.
Leader of the Goonswarm Economic Warfare Cabal.
Creator of Burn Jita
Vile Rat: You're the greatest sociopath that has ever played eve.
|
Freelancer117
So you want to be a Hero
352
|
Posted - 2015.10.14 15:45:24 -
[78] - Quote
Gratz on moving to DDR4
Your new server cpu's are over a year on the market and mid range, hope you got a good price.
source: http://ark.intel.com/products/family/78583/Intel-Xeon-Processor-E5-v3-Family#@All
You mentioned Eve Forever and Dust514 wil run on Eve proxies, is CCP games still lasor focused on tying Dust514 and New Eden capusleers together in A Future Vision with enhanced interactions ?
Regards, a Freelancer
The players will make a better version of the game, then CCP initially plans.
http://eve-radio.com//images/photos/3419/223/34afa0d7998f0a9a86f737d6.jpg
The heart is deceitful above all things and beyond cure. Who can understand it?
|
|
CCP DeNormalized
C C P C C P Alliance
306
|
Posted - 2015.10.14 15:56:25 -
[79] - Quote
CPU's are E7-8893 v3, not E5
http://ark.intel.com/products/84688/Intel-Xeon-Processor-E7-8893-v3-45M-Cache-3_20-GHz
Launch Date Q2'15
Errr, at least the DB CPU's are :) I don't really care so much about the others :)
CCP DeNormalized - Database Administrator
|
|
Freelancer117
So you want to be a Hero
354
|
Posted - 2015.10.14 16:01:11 -
[80] - Quote
Hehe, wouldn't mind switching my I7 cpu to one of those !
The players will make a better version of the game, then CCP initially plans.
http://eve-radio.com//images/photos/3419/223/34afa0d7998f0a9a86f737d6.jpg
The heart is deceitful above all things and beyond cure. Who can understand it?
|
|
Josia
Sunrise Services
2
|
Posted - 2015.10.14 18:31:45 -
[81] - Quote
Can you run Minecraft on it? |
Luca Lure
Obertura
48
|
Posted - 2015.10.14 19:32:36 -
[82] - Quote
Clearly EVE is dying. Hamsters need new cages.
GÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇòGÇò
The essence of the independent mind lies not in what it thinks, but in how it thinks.
|
Indahmawar Fazmarai
4030
|
Posted - 2015.10.14 20:05:15 -
[83] - Quote
Out of curiosity... where will come from all the additional players needed to use/justify such powerful hardware?
CCP Seagull: "EVE should be a universe where the infrastructure you build and fight over is as player driven and dynamic as the EVE market is now".
62% of players: "We're not interested. May we have Plan B, please?"
CCP Seagull: "What Plan B?"
|
Corraidhin Farsaidh
Farsaidh's Freeborn
1745
|
Posted - 2015.10.14 21:02:55 -
[84] - Quote
The others matter?
|
|
CCP Gun Show
C C P C C P Alliance
14
|
Posted - 2015.10.14 21:07:34 -
[85] - Quote
Aryth wrote:In the writeup I don't see why IBM. Did you bake these off against UCS and they were faster? Or just whitebox.
Maybe your performance needs are very niche but I have yet to see a bakeoff where IBM won against almost anyone.
We did an intensive comparison with couple of vendors but came to this conclusion out of couple of reasons
This is a vague answer and does not tell you much apart from that we did our du diligence
Plus our relationship with the Icelandic vendor is excellent after decade of cooperation, I literally can call the lead IBM SAN expert anytime 24/7 and they are quick to support us in a critical scenario with good escalation path into IBM
Hope this answer helps and please keep on asking about TQ Tech III |
|
|
CCP Gun Show
C C P C C P Alliance
14
|
Posted - 2015.10.14 21:09:34 -
[86] - Quote
Corraidhin Farsaidh wrote:
Oh yes they do ! The entire cluster matters
Demoralized is just lazer focused on the DB machines apparently |
|
virm pasuul
FRISKY BUSINESS. No Handlebars.
319
|
Posted - 2015.10.14 22:20:57 -
[87] - Quote
What's the warranty on the new kit? Default 3 year or extended?
|
|
CCP DeNormalized
C C P C C P Alliance
307
|
Posted - 2015.10.14 22:27:23 -
[88] - Quote
CCP Gun Show wrote:Oh yes they do ! The entire cluster matters CCP Denormalized is just lazer focused on the DB machines apparently
I suppose I care about the rest as well... If not for those others my shiny DB servers would just sit idle all day long :)
CCP DeNormalized - Database Administrator
|
|
Disco Dancer Dancing
State War Academy Caldari State
0
|
Posted - 2015.10.14 22:51:29 -
[89] - Quote
Being someone that builds complex, large datacenters for both private and public use on a rather period basis, I'm not that impressed on the path of physical architecture that you are looking at. For whatever reason, why are you looking at a traditional, silo-based solution with storage and compute in different silos and traversing a "slow" FC link to make it work when not hitting the cache in RAM. Any particular reason why you are not looking on a more modern, flexible and scalable platform then the one described in the blogpost?
Seeing that everything not in the RAM-cache have to traverse the FC switch we can quickly give a few numbers on the actual latency and round-trip on several different ways of accessing data on different locations
L1 cache reference 0.5 ns Branch Mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1KB with Zippy 3,000 ns Sent 1KB over 1Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD 150,000 ns 0.15 ms Read 1MB sequentially from memory250,000 ns 0.25 ms Round trip within datacenter 500,000 ns 0.5 ms Read 1MB sequentially from SSD 1,000,000 ns 1 ms, 4x memory Disk seek 10,000,000 ns10 ms, 20x datacenter round trip Read 1MB sequentially from disk 20,000,000 ns20 ms, 80x memory, 20x SSD Send packet CA -> Netherlands -> CA150,000,000 ns150 ms
Looking at the figures, as soon as we start to traverse several layers we add up latency on the whole request, if we need to traverse the FC, to hit the storage nodes, then hit the disk and back, latency can rather quickly add up. Keeping the data as local as possible is the key, mainly in Memory, or as close to the node as possible without traversing the network (Sure, FC is stable, proven and gives a rather low latency, but from other standpoint you could argue that it is dead in the upcoming years as we are moving towards utilizing RDMA over Converged Fabrics or the like).
On another note, if we look on a mainstream enterprise SSD we can find a few figures: 500MB/s Read and 460MB/s Write If we put these into the following calculation to see when we saturate a traditional storage network: numSSD = ROUNDUP((numConnections * connBW (in GB/s))/ ssdBW (R or W))
We get the following table: Network BW SSDs required to saturate network BW Controller ConnectivityAvailable Network BWRead I/OWrite I/O Dual 4Gb FC 8Gb == 1GB 2 3 Dual 8Gb FC 16Gb == 2GB 4 5 Dual 16Gb FC 32Gb == 4GB 8 9 Dual 1Gb ETH 2Gb == 0.25GB 1 1 Dual 10Gb ETH 20Gb == 2.5GB 5 6
This is without taking into account the roundtrip to access the data, and is counting with unlimited CPU power as this can also become saturated. We can see that we don't need that many SSD to saturate a network. Key point here, try and keep the data as local as possible, once again not traversing the network with the added latency and network limitation.
We can also do calculations on difference on hitting per say a local storage cache in the memory, or hitting a remote storage cache (Per say, SAN controller with caching). I know from the top of my head which are the fastest, key point, once again, keep the data as local as possible.
There are several technologies pin-pointing these issues that are seen in traditional silo datacenters, have you looked at any, and if so what is the reason that these do not fit your needs?
|
|
CCP Gun Show
C C P C C P Alliance
16
|
Posted - 2015.10.15 00:05:47 -
[90] - Quote
Disco Dancer Dancing wrote:Being someone that builds complex, large datacenters for both private and public use on a rather period basis, I'm not that impressed on the path of physical architecture that you are looking at. For whatever reason, why are you looking at a traditional, silo-based solution with storage and compute in different silos and traversing a "slow" FC link to make it work when not hitting the cache in RAM. Any particular reason why you are not looking on a more modern, flexible and scalable platform then the one described in the blogpost?
Seeing that everything not in the RAM-cache have to traverse the FC switch we can quickly give a few numbers on the actual latency and round-trip on several different ways of accessing data on different locations
L1 cache reference 0.5 ns Branch Mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1KB with Zippy 3,000 ns Sent 1KB over 1Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD 150,000 ns 0.15 ms Read 1MB sequentially from memory250,000 ns 0.25 ms Round trip within datacenter 500,000 ns 0.5 ms Read 1MB sequentially from SSD 1,000,000 ns 1 ms, 4x memory Disk seek 10,000,000 ns10 ms, 20x datacenter round trip Read 1MB sequentially from disk 20,000,000 ns20 ms, 80x memory, 20x SSD Send packet CA -> Netherlands -> CA150,000,000 ns150 ms
Looking at the figures, as soon as we start to traverse several layers we add up latency on the whole request, if we need to traverse the FC, to hit the storage nodes, then hit the disk and back, latency can rather quickly add up. Keeping the data as local as possible is the key, mainly in Memory, or as close to the node as possible without traversing the network (Sure, FC is stable, proven and gives a rather low latency, but from other standpoint you could argue that it is dead in the upcoming years as we are moving towards utilizing RDMA over Converged Fabrics or the like).
On another note, if we look on a mainstream enterprise SSD we can find a few figures: 500MB/s Read and 460MB/s Write If we put these into the following calculation to see when we saturate a traditional storage network: numSSD = ROUNDUP((numConnections * connBW (in GB/s))/ ssdBW (R or W))
We get the following table: Network BW SSDs required to saturate network BW Controller ConnectivityAvailable Network BWRead I/OWrite I/O Dual 4Gb FC 8Gb == 1GB 2 3 Dual 8Gb FC 16Gb == 2GB 4 5 Dual 16Gb FC 32Gb == 4GB 8 9 Dual 1Gb ETH 2Gb == 0.25GB 1 1 Dual 10Gb ETH 20Gb == 2.5GB 5 6
This is without taking into account the roundtrip to access the data, and is counting with unlimited CPU power as this can also become saturated. We can see that we don't need that many SSD to saturate a network. Key point here, try and keep the data as local as possible, once again not traversing the network with the added latency and network limitation.
We can also do calculations on difference on hitting per say a local storage cache in the memory, or hitting a remote storage cache (Per say, SAN controller with caching). I know from the top of my head which are the fastest, key point, once again, keep the data as local as possible.
There are several technologies pin-pointing these issues that are seen in traditional silo datacenters, have you looked at any, and if so what is the reason that these do not fit your needs?
Wow that's one serious question right there , hope you will come to fanfest 2016 to talk about latency
I just saw this on my mobile and allow me to get you a proper answer in a day or two
Excellent stuff !
|
|
|
|
|
|
Pages: 1 2 [3] 4 :: one page |
First page | Previous page | Next page | Last page |