cmaxx
   |
Posted - 2010.06.21 16:28:00 -
[1]
Hey CCP Yokai. I'm really interested in what you're doing and I wish you guys all the best on Wed.
I am interested, and a little surprised, by the comment that you've not saturated 1 GbE yet - cos that's not what I'd be paying attention to. :)
From my perspective, looking at the saturation of a network when considering whether to move up to the next speed grade means missing a big opportunity for latency reduction/management.
Consider: a server like a SQL Server or a SOL server won't be able to do anything at all with a packet until the very last byte has been received off the wire. Only then can the payload checksum be verified and the data within it passed up through the network stack, across the kernel boundary to the process. (aside: If it's a fragmented packet it'd be even worse.)
Now, consider the respective times for the *last* byte of a packet to be delivered over 1 GbE and 10 GbE.
The 10GbE connexion cuts that last-byte latency 10-fold. The first bit of the first byte should get there about as fast in both cases, but it's the last bit of the last byte that lets work begin.
Consider also that servers receive packets in streams and bursts, from lots of clients.. so that 10-fold reduction in the time-to-last-byte for one packet leads to a 19-fold reduction in latency for the next packet in line, and so on out until you reach the end of an average burst pattern.
And you see the benefit in both directions, from client to server, and from server back to client.
For my part, I think it's really worth bumping the network speed grade when the latency seen at end-user clients has a significant component made up from the sequential and/or combinatoric latency terms within the server cluster, which is my best guess for how TQ ends up working today.
10GbE port costs are now quite affordable, as are client card costs, especially if you get a bulk discount.
Now Cisco's latest hardware may be quite good but being a corporate behemoth and the market leader, they're frankly a little complacent, a little interally disorganized and no longer the technical be-all-and-end-all they once were. I have a little insight here.
With TQ being such a small (numerically) cluster and having important latency-driven behaviour within the server cluster architecture, I'd consider looking for switches from someone who has crafted their 10GbE hardware for the perfect combination of aggressive performance (even sub-microsecond) and high reliability.. the sort of thing people would use for high frequency trading infrastructure in global financial exchanges and for serious scientific computing. Someone like Arista: http://www.aristanetworks.com/ . Turns out they're also very price-competitive. Potnential for wins all round.
Enable Chimney etc. on Windows to get the best possible assist from the hardware and it'll push the bottlenecks right back to the software, CPUs and memory subsystems again.
But I guess you've spent your budget, so.. maybe in another year or so. :P
|