| Pages: 1 [2] 3 4 :: one page |
| Author |
Thread Statistics | Show CCP posts - 6 post(s) |
Niccolado Starwalker
Shadow Templars
   |
Posted - 2009.04.30 11:59:00 -
[31]
Edited by: Niccolado Starwalker on 30/04/2009 11:59:29
 Originally by: Akima Jarka
Edited by: Akima Jarka on 29/04/2009 20:25:53
I didn't understand 1/2 that blog, but I think your saying something good happened.
So, Good Job!
This 
It was still a very good blog!
Originally by: Dianabolic
Your tears are absolutely divine, like a fine fine wine, rolling down your cheeks until they flow down the river of LOL |
rowbin hod
Brutor tribe
|
Posted - 2009.04.30 12:03:00 -
[32]
Originally by: CCP cNOC
There is another way to get closer to the end user ISP's, we are sitting in the same building as LINX. This is one of the largest "free" peering hub in the world, a meet-me point for many of the ISP's. This is something we are looking into and could be the next step for us. But I think the quality now of the Tier1 connections is that good, and we are mostly now seeing over congested lines from the end user ISP's to the Tier1's being the bottleneck. That unfortunately is out of our reach.
I would say that joining LINX is definitely the best way to go. I work for a London-based ISP (99% content serving as opposed to providing web access for people) pushing 1.6Gbit/sec (small change really!) and well over 50% of our traffic goes over LINX. Not only is this free (aside form the port charge), but it reduces reliance on one or two providers and their BGP sessions to you and their own connections within their network.
I'm slightly disappointed to see Packet Exchange being used, not having had very good experiences with them in the past. London is awash with BGP providers who deliver excellent service, so I'd like to thisnk that if there are any problems with Packet Exchange then CCP will be able to move over to a shiny new provider pretty quickly!
|
Commander Azrael
Three Shades of Brown
|
Posted - 2009.04.30 13:47:00 -
[33]
Originally by: rowbin hod
I would say that joining LINX is definitely the best way to go. I work for a London-based ISP (99% content serving as opposed to providing web access for people) pushing 1.6Gbit/sec (small change really!) and well over 50% of our traffic goes over LINX. Not only is this free (aside form the port charge), but it reduces reliance on one or two providers and their BGP sessions to you and their own connections within their network.
Agreed, A quick look at our BGP neighbor listing on our edge routers recieving prefixes via linx shows around 150 neighbours, all for the price of the LINX port! bargain really considering how many extra people you can extend your peering service to and how much bandwidth we shove over it
Here's to a faster connection to the eve cluster 
|
T'san Manaan
|
Posted - 2009.04.30 18:18:00 -
[34]
Ohhh......
So your saying the internet space ships get from iceland to my computer via magic.
Got it.
|
Lady Doreen
Gallente
|
Posted - 2009.04.30 18:30:00 -
[35]
Very well done CCP
problem analysis, hardware platform selection etc ...
And yes you need to have a good understanding of
how routing works, to get the most out of internet
for your customers.
Nice that you are still giving this very good information...
And kudos ... that were some $bucks you needed to invest.
Thanks a lot for doing this for your customers
|
Gnulpie
Minmatar
Miner Tech
|
Posted - 2009.04.30 18:42:00 -
[36]
THAT I call a good devblog.
Well done.
|
Anthal
The Warp Squad
|
Posted - 2009.04.30 20:35:00 -
[37]
Ohhh, I understood every word of that. Probably because I do it for a living, but still. That's some nice gear, wish my boss would let our company upgrade our routers and switches on a whim! Out of curiosity, with all of those peers, how much bandwidth is available, and how much are you using?
It's also nice to see that your NOC employees actually know what a traceroute is and how it works. For everyone else who's eyes are glazing over, this is a fantastic read: http://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N45.pdf
|
Komen
Gallente
Aliastra
|
Posted - 2009.04.30 22:46:00 -
[38]
I'm an...informed layman...when it comes to interwebz 'how does it work' but I got most of that.
Thank you for the informative blog, and also for pursuing every single avenue possible to reduce lag.
We appreciate your diligence.
|
Rin Marcosad
Gallente
Pillowsoft
|
Posted - 2009.05.01 00:53:00 -
[39]
I love this behind-the-scenes stuff. It's cool to see the real issues and solutions that are hidden away in the background of running a game like EVE!
|
static zero
Minmatar
Power of the Phoenix
|
Posted - 2009.05.01 04:10:00 -
[40]
Awesome stuff. I love hearing about the nuts and bolts. Thank you very much for posting!
<nit action="pick"> BGP can provide some amount of control over return traffic, mainly through as-path manipulation but also through BGP communities. However, it's still up to each ISP whether to accept the announcer's request. So it is largely non-deterministic anyway. </nit>
Sounds like you're quashing latency well by becoming more broadly connected. I've noticed the performance on my US Comcast link has been better as of late. I wonder how many other providers you can reach with that IX port.
As long as you're contemplating IX membership, I'd recommend checking out the UK Network Operators' Forum. It's been a helpful resource for me, even managing connectivity for a small datacenter in London.
Moreover, with all those new links, it may be worth it for CCP to obtain provider-independent space. RIPE NCC permits /24 or larger PI allocations, from what I remember. Given that TQ is served from a contiguous /23, and that TQ is becoming highly connected, I'm certain you'd qualify.
-Static
-static zero |
Masquitar
Minmatar
High House Of Shadows
|
Posted - 2009.05.01 10:47:00 -
[41]
Nice blog! Thanks for putting us in the loop on this.
|
Bom Bommer
|
Posted - 2009.05.01 12:45:00 -
[42]
A very interesting blog, imho. First of all, because I always enjoy technical stuff and networking is one of those fields where I am I the most ignorant. Second of all, I was actually suspecting there could be routing problems behind that silly lag. So I was glad to get an insightful view of the things on your side and what you were able to do to overcome them. Finally, I think you put a good balance between the technical stuff and comprehensibility. My only regret is that you could have provided one or two diagrams or, on some topics, links to Wikipedia or others : you need to know the appropriate keywords to search things by yourself.
|
Iyotaka
Iyotaka Union
|
Posted - 2009.05.01 13:30:00 -
[43]
A pleasure to read.
Sounds like an excellent solution. And I learned a few things I was not aware of - never having had to really worry about how BGP communicates e.g..
|
Chainsaw Plankton
IDLE GUNS
IDLE EMPIRE
|
Posted - 2009.05.01 16:20:00 -
[44]
Originally by: Chribba
packet pron.
Stop sniffing my packets!!!!
 errm, I think he means something else by that
|
Arganato
|
Posted - 2009.05.01 16:49:00 -
[45]
Nice Dev-blog and keep up the good work :D
|
Taram Caldar
Noir.
|
Posted - 2009.05.01 20:36:00 -
[46]
Edited by: Taram Caldar on 01/05/2009 20:37:05
Nice blog!
However the author made a small technical mistake.
The internet didn't start out as universities talking to each other. It started out as a few US military and research sites talking to each other and was named the arpanet ;) It grew to include other gov't sites and institutions of higher learning shortly thereafter and became the internet.
Sorry.... as someone who's been deeply involved with the internet since the 80's that's a pet peeve of mine :)
|
|
CCP cNOC
|
Posted - 2009.05.01 23:47:00 -
[47]
Edited by: CCP cNOC on 01/05/2009 23:53:42
yeah, well we all know the story (somewhat ;), I mostly stick by the university story as it's how I came first in contact with the "internet" as we know it today, and it's was mostly just universities that where active at that time... But BGP actually comes much later, See the RFC 1105 is dated June '89. Perhaps someone here knows when BGP became the defacto routing protocol of the internet. BGP v4 as we use today is dated '95
sorry for being rough on the history, go USAF ! 
|
|
G'Kar5
Gallente
Intaki Research and Manufacturing
|
Posted - 2009.05.02 02:29:00 -
[48]
Edited by: G''Kar5 on 02/05/2009 02:31:05
Decent writeup, but BGP isn't really that scary and you will never learn it from a book. Check out the different BGP communities providers will accept to do various things. A lot of the providers will do selective prepends, allow you to treat you as non-transit (ie send your routes to their customers only), let you set local-pref, etc. Check out the following URL: http://www.onesc.net/communities/ They have a list of most communities that providers accept. Most only apply if you are a direct customer of theirs though. By using things like this and playing prepend games you can kind of control the return path as well.
I would also check out Dave Meyer's UO route-views project to get a good view of how the world sees your routes: http://routeviews.org/ . You can also just telnet into their route servers and just do a show ip bgp 87.237.38.0. Its very useful to see how big chunks of the Internet react to you prepending and playing various community games.
Have you guys done any netflow analysis of your traffic to figure out which ASes most of your users are in? If not, I would definitely turn on netflow on and take a peek. Its a very good tool for figuring out who you need to peer with. If you haven't, you can snag a good freeware collector from http://www.splintered.net/sw/flow-tools/ And the good thing is that pretty much everybody is in Telecity-1/Telecity-2.
As always, if you need free expert advice on anything, reach out to your user base. We are geeks. I know very little about cluster computing, but I used to live on Tier-1 core routers 24x7 but now work for that Cisco company. Oh yea you should look this crazy Swede up, he's fun to hang out with: http://www.youtube.com/watch?v=olbGB-zkNr0
|
To'Shua
|
Posted - 2009.05.02 04:36:00 -
[49]
Originally by: CCP cNOC
Edited by: CCP cNOC on 01/05/2009 23:53:42
yeah, well we all know the story (somewhat ;), I mostly stick by the university story as it's how I came first in contact with the "internet" as we know it today, and it's was mostly just universities that where active at that time... But BGP actually comes much later, See the RFC 1105 is dated June '89. Perhaps someone here knows when BGP became the defacto routing protocol of the internet. BGP v4 as we use today is dated '95
sorry for being rough on the history, go USAF !
Documenting history is important, even for computer-related stuff. Presenting the beginning of the story as if it started with universities is a denial of the complete history and does a disservice to younger generations. There were reasons the Defense Dept. wanted something like ARPAnet, and the resulting "internet" is largely a success when considered from the perspective of the original goal. What is perhaps "shocking" is what happened with the "internet" once it was let loose from the labs. The speed and impact of the "internet" on world societies has been truly phenomenal -- I dare say rivaling that of regional and global pandemics and wars. Eve-Online itself is one small part of that continuing story.
|
rValdez5987
Amarr
Imperial Guard.
|
Posted - 2009.05.02 19:09:00 -
[50]
Edited by: rValdez5987 on 02/05/2009 19:11:22
Oh god you can tell I'm a Cisco engineer when I get a hard on over blogs such as this one :P
So far from reading, Cogent is actually the Tier 1 provider for the isp that I currently use (Which is a tier 3 ofc)
reserved* for rest of comments... |
oldfrogger's broda
|
Posted - 2009.05.02 19:32:00 -
[51]
Edited by: oldfrogger''s broda on 02/05/2009 19:33:51
It was a good idea to move to rsp720-3BXLs
Don't forget - you have to cope with two issues
1. BGP
2. lots of "uncommon" packets (you datacentre - like datacentre of a search portal produces smth very uniqe)
all you will need in nearest future - 4gb mem upgrade MEM-RSP720-4G= (when or just 1.5 months (5-6 weeks)before the moment you feel that there is not enough mem for all that fullviews you want)
and to install modules like WS-X67... with DFC3CXL ( ... dont have to burn money with 7600-ES20...3CXL until you decide that you want to do some really wierd stuff with packets ) ... anyway - ask for live tests (Cisco has demo equipment stock =)
Nexus is too fresh ... if Cisco wants it in your system
insist on lab tests with artificail traffic patterns equal to yours BEFORE implementing (to avoid thing you hit with 32 sups)
ASR is for ISPs traffic aggregation (not your case)
CRS - you just dont need it until you do not have several ASR's to have traffic from
12000 forget about it ... ("old" CRS)
P.S: I think there are lots or Routing CCIE's playing EVE ... ask them for free advice
|
Carreira
|
Posted - 2009.05.02 21:37:00 -
[52]
Any idea when this "We will soon turn on peering with 120 smaller SP's through a multilateral peering platform that PacketExchange operates." will be hapening?
I am (still) suffering from connection issues to TQ (socket closed) due to a hop in Lisbon.
If I understand correctly, once these 120 smaller peerings are turned on, the problems should dissipate?
|
|
CCP cNOC
|
Posted - 2009.05.02 23:35:00 -
[53]
Edited by: CCP cNOC on 02/05/2009 23:45:14
Edited by: CCP cNOC on 02/05/2009 23:40:30
Edited by: CCP cNOC on 02/05/2009 23:36:42
Originally by: Carreira
Any idea when this "We will soon turn on peering with 120 smaller SP's through a multilateral peering platform that PacketExchange operates." will be hapening?
I am (still) suffering from connection issues to TQ (socket closed) due to a hop in Lisbon.
If I understand correctly, once these 120 smaller peerings are turned on, the problems should dissipate?
We where not 100% happy how this was configured, and we have the solution in hand and very soon I will start using that path. (perhaps Monday DT). I will ask around and find your IP and look if we can find a better path to you. The "socket closed" issue is something we are very much monitoring..
Bit on the hardware. Well, Not to give out all the details of our hardware, you now know what supervisor we are using for the edge routers ;P so, why not some more ;P Yes we are using 4Gb of RAM..( Anyone know what Blizzard is doing ? ;)
The internet routing table is around 280k routes, We are using a /23 prefix for TQ. Few years back we'd probably would have to use full /21 or perhaps get away with /22, but now it's no problem sending out /24 prefix without much filtering. The number of small prefixes is quite amazing, and it looks like most of the big networks stopped filtering on small prefixes...
I dont think Cogent is an tier1 provider, only a transit.. but they are very big, correct me if I'm wrong.. but Tier1 = connected everywhere to the internet, so they don't have to buy any transit service to get to some parts of the internet.. We are just sitting close to the tier1, but we don't supply any peering to other customers, so we are not part of the tiered model in that sense.
|
|
Levanel
|
Posted - 2009.05.02 23:47:00 -
[54]
Whatever you're doing, it's still not helping.
|
ju4n1ta
Gallente
The Scope
|
Posted - 2009.05.03 09:56:00 -
[55]
I love this type of g33ky blogs!! Keep 'em coming!
|
Inanna Sumer
GoonFleet
GoonSwarm
|
Posted - 2009.05.04 03:14:00 -
[56]
Originally by: CCP cNOC
The internet routing table is around 280k routes, We are using a /23 prefix for TQ. Few years back we'd probably would have to use full /21 or perhaps get away with /22, but now it's no problem sending out /24 prefix without much filtering. The number of small prefixes is quite amazing, and it looks like most of the big networks stopped filtering on small prefixes...
A lot of providers will accept longer (than /24) prefixes from customers and will carry them internally- and even send them to their own customers, however these will typically be filtered out at their peering adjacencies.
Memory isn't really that critical once you go past 2GB or so, 1.2M BGP path entries (~ 4 full feeds) only takes 60MB of memory, and scales fairly linearly.
If you guys do eventually want to allow users to select their outbound path through a portal, you'll want to run your own custom BGP daemon on a server somewhere (bgpd.pl or similar) that injects /32 routes (into your edge/upstream routers would probably be best, assuming they're setting nexthop-self toward your distribution layer) into your BGP RIB with a nexthop of the desired transit provider, just make sure you keep an eye on your TCAM utilization if you end up doing something like that since the default TCAM assignment on the RSP720 only allows for 512M IPv4 prefixes- and with the growing rate of prefixes in the IPv4 DFZ that could get cramped in year or four if you have an additional 40k /32s in there for TQ user traffic engineering.
|
G'Kar5
Gallente
Intaki Research and Manufacturing
|
Posted - 2009.05.04 04:56:00 -
[57]
Edited by: G''Kar5 on 04/05/2009 04:57:06
Originally by: CCP cNOC
Edited by: CCP cNOC on 02/05/2009 23:45:14
Edited by: CCP cNOC on 02/05/2009 23:40:30
Edited by: CCP cNOC on 02/05/2009 23:36:42
Originally by: Carreira
Any idea when this "We will soon turn on peering with 120 smaller SP's through a multilateral peering platform that PacketExchange operates." will be hapening?
I am (still) suffering from connection issues to TQ (socket closed) due to a hop in Lisbon.
If I understand correctly, once these 120 smaller peerings are turned on, the problems should dissipate?
We where not 100% happy how this was configured, and we have the solution in hand and very soon I will start using that path. (perhaps Monday DT). I will ask around and find your IP and look if we can find a better path to you. The "socket closed" issue is something we are very much monitoring..
Bit on the hardware. Well, Not to give out all the details of our hardware, you now know what supervisor we are using for the edge routers ;P so, why not some more ;P Yes we are using 4Gb of RAM..( Anyone know what Blizzard is doing ? ;)
The internet routing table is around 280k routes, We are using a /23 prefix for TQ. Few years back we'd probably would have to use full /21 or perhaps get away with /22, but now it's no problem sending out /24 prefix without much filtering. The number of small prefixes is quite amazing, and it looks like most of the big networks stopped filtering on small prefixes...
I dont think Cogent is an tier1 provider, only a transit.. but they are very big, correct me if I'm wrong.. but Tier1 = connected everywhere to the internet, so they don't have to buy any transit service to get to some parts of the internet.. We are just sitting close to the tier1, but we don't supply any peering to other customers, so we are not part of the tiered model in that sense.
Cogent is a pseudo-tier1. Getting into a debate about what is and what is not a tier-1 ISP is going to turn into a mess. Key terms: Settlement free peering, transit, and non-tranist. Settlement free peering basically means that two providers connect with each for mutual benefit and neither side pays the other although they equally pay for the cost of the interconnect (ie POS/DWDM backhaul, colo interconnect fees, etc). Transit is what CCP is doing, ie paying somebody to send you the global internet routing table. Non-transit (or paid peering) is paying a provider for them to only send you their customer routes. Non-transit and settlement free peering are EXACTLY the same from a BGP perspective, the only difference in non-transit one party pays for the service. This is what makes telling the difference between a tier-1 and somebody else hard. Often times when somebody buys non-transit, part of the contract includes something like both sides claim they peer with each other. Anyway, I digress. Tier-1 means every BGP session is settlement free or a customer; you pay nobody for routes.
ATT, Sprint, Level3, Verizon Biz, NTT/Verio, Qwest, Global Crossing, and Savvis are true tier-1's. Cogent and AOL are may or may not be tier-1. Some of the big guys have tried de-peering them lately. Not sure of the latest status there. People like BT, DT, FT, Telia, Reach, and Telstra are very close to a tier-1 in the sense that they probably don't buy transit from anybody but they pay for non-transit service from at least one of the true tier-1's. I might have missed some of the other pseudo-tier-1's though.
Anyway, "tier-1" has largely changed to mean "big honking network" as opposed to "I don't pay anybody for transport"
Edit: BTW pretty much everybody will accept a /24. If you announce a /24 and your upstream accepts it, it will propagate to all routers on the internet.
|
G'Kar5
Gallente
Intaki Research and Manufacturing
|
Posted - 2009.05.04 05:10:00 -
[58]
Edited by: G''Kar5 on 04/05/2009 05:14:40
Apparently I ran out of space.
I wanted to also let you know that being close to the tier-1's isn't necessarily in CCP's best interest. All the Tier-1's pretty much peer with themselves and they don't let anybody new into their club. This has caused the tier-2's to "peer around the core" and create the "doughnut" effect. The tier-2's send A LOT of traffic to each other, but they don't want to pay the tier-1's to deliver that traffic. So they have all peered with each other and exchange most of their traffic through Settlement Free peering. The Tier-2's only use the tier-1's (ie buy transit) to get to places where they don't have enough direct traffic to justify the cost of peering with that provider. Remember, peering isn't free you still have to buy router ports, backhaul, colo power/space, etc.
Now, CCP's customers all connect to residential ISP's. Most of these ISP's are not tier-1's (Verizon Business, Qwest, and ATT being the exception). What this means is that if CCP had multiple 10GE's with every tier-1 (and only that upstream bandwidth) the service may still suck. This is because CCP is relying on the fact that the ISP that services CCP's customers (ie home users) has good connectivity with the tier-1's. This is often not true. The tier-2's HATE to pay the tier-1's for bandwidth and they often run those links as hot as they possibly can before upgrading them.
The best thing to do is to directly peer with as many tier-2 (ie residential ISP's) as you possible can. What you should do is enable netflow and start categorizing the amount of traffic (not users) based on origin AS. Pick the top 10 and start trying to peer with them (most should probably be in Telecity-1/2). The tricky part is getting them to agree as CCP probably isn't on their list. The residential ISP's do the same thing, except they have people like Google (Youtube) and Yahoo at the top of their list. If you can get them to agree that you have enough traffic to justity it, they will peer with you. You could also try a "get out the NOC call campaign". By this I mean the bane of any residential ISP is a customer service call. If CCP is having trouble with a ISP, and say 2000 Eve-Online players call their ISP every day to complain about reaching the Eve servers the ISP might just peer with CCP. It will cost the ISP way more to deal with the phone calls than to just toss you a GigE or so especially if they are already colocated with you.
|
london
Gallente
Dark-Rising
|
Posted - 2009.05.04 05:11:00 -
[59]
jesus christ, this is the most esoteric blog ever.
i think my eyes fell out.
|
Inanna Sumer
GoonFleet
GoonSwarm
|
Posted - 2009.05.04 15:59:00 -
[60]
Edited by: Inanna Sumer on 04/05/2009 15:59:04
Originally by: G'Kar5
Edit: BTW pretty much everybody will accept a /24. If you announce a /24 and your upstream accepts it, it will propagate to all routers on the internet.
Correction: Most routers on the internet.
If you announce a /24 without it being part of a covering /23 or /22 you should expect to get filtered by some smaller operators who are still running older hardware which only allows up to 256k entries in the FIB (such as people running Sup32s and Sup2s on 6500/7600 platforms). I'd hope they're all running with default routes though.
|
| |
|
| Pages: 1 [2] 3 4 :: one page |
| First page | Previous page | Next page | Last page |