CenturyLink routes EVERYTHING out to Chicago, IL for Ohio customers.. Even LOCAL traffic >:(
To talk to neighbors on a different ISP, CenturyLink routes us all the way out to Chicago and back, adding latency, nasty! They do ZERO peering in Columbus, Ohio with any big Tier 1 ISPs or any of the competitor ISPs that serve the Ohio area, a bad practice I must say. Yet they claim they have a POP in Columbus and Lewis Center (near by) yet I don’t see them using it to peer with any competitor ISPs. I’ve known about this problem, but had to post this as I’ve gotten sick of it. Here is CenturyLink’s footprint for Ohio, as you can see they offer services in a relatively large portion of Ohio, mostly rural areas.

I’ve kept the Vice President of Operations for the Ohio market, Mr. Michael Brill in the picture so you know who to contact, which I’ll be e-mailing him this post as well. 😉 As you can also tell from the photo, they show core fiber routes going to Columbus and stating they have a data center there. Lets see how good their peering is in Ohio. 🙂

Time Warner Cable (now “Spectrum” Charter Communications)

This is the king ISP in Ohio. If you live in a city or town where there’s decent enough house density, you have Time Warner Cable for Internet. They are everywhere, that it’s literally disgusting the monopoly they have on Ohio. If you’re an ISP in Ohio, or have a data center in Ohio, you DEFINITELY want to peer with TWC, as it’ll benefit the MAJORITY of Ohio customers. They are the 2nd biggest cable ISP in the US, why wouldn’t you peer with them? CenturyLink peers with the 1st biggest cable ISP Comcast, but neglect TWC? Also, TWC was bought by Charter Communications, making it’s combined ISP, Spectrum, and even bigger ISP. You’d think that’d be an incentive to peer with them.. Here’s TWC coverage map, pay attention to how much they cover Ohio…

If you’re offering Internet services in the state of Ohio, you are very foolish if you don’t peer with Time Warner Cable.. My suggestion to CenturyLink is to peer with them in Columbus at least.

Lets see how CenturyLink DSL in Mount Vernon, Ohio routes to Time Warner Cable’s main router in Mount Vernon,

traceroute to 24.33.161.129 (24.33.161.129), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.616 ms 0.742 ms 0.630 ms
2 65.131.220.254 (65.131.220.254) 8.721 ms 9.069 ms 10.253 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 10.900 ms 12.007 ms 13.117 ms Mansfield, OH
4 chp-brdr-04.inet.qwest.net (67.14.8.238) 36.192 ms 38.055 ms 39.188 ms Chicago, IL
5 206.111.2.153.ptr.us.xo.net (206.111.2.153) 40.339 ms 40.987 ms 42.393 ms
6 vb2001.rar3.chicago-il.us.xo.net (207.88.13.130) 32.039 ms 33.237 ms 33.142 ms
7 207.88.13.7.ptr.us.xo.net (207.88.13.7) 41.886 ms 30.084 ms 30.477 ms
8 216.3.52.42 (216.3.52.42) 19.128 ms 20.583 ms 21.692 ms
9 unk-426d05e0.adelphiacom.net (66.109.5.224) 41.869 ms 36.894 ms 37.886 ms
10 bu-ether11.chctilwc00w-bcr00.tbone.rr.com (66.109.6.21) 38.231 ms 43.265 ms 46.841 ms TWC’s Chicago link, they don’t even peer in Chicago, going through XO
11 be2.clmkohpe01r.midwest.rr.com (66.109.6.69) 40.800 ms 44.656 ms 46.731 ms Ohio
12 be10.clevohek01r.midwest.rr.com (65.25.137.134) 56.566 ms 56.444 ms 38.287 ms
13 agg1.clmcohib01r.midwest.rr.com (65.29.1.39) 43.634 ms 43.419 ms 49.167 ms
14 agg1.mtvroh2702h.midwest.rr.com (24.33.161.129) 65.363 ms 66.102 ms 66.108 ms

WOW, that’s a lot of hops to go from Mount Vernon, Ohio to Mount Vernon, Ohio. See the problem there? To Skype or play XBOX Live, etc. with our neighbors on Time Warner we’ll see ~65+ms, that’s disgusting! My brother was playing xbox against his friend in town and was lagging badly because the packets bouncing all the way to Chicago and back. You can play with other people in other states with less latency as you don’t have to bounce back and forth.

OARnet

This is a 100Gbps fiber network that links multiple big cities in Ohio for local government, education, research, etc. A lot of Universities, K-12 schools, state government, local county government, hospitals, etc. use OARnet. What’s funny is CenturyLink is a BIG partner in this and allegedly peers and does a lot of the OARnet network.

When we look at OARnet’s ASN peers, we see AS209, Qwest Communications, which is CenturyLink’s main AS as a peer.

So you’d think this would happen in Columbus, Ohio and CenturyLink customers would have nice low-latency to colleges, schools, government in Ohio, right? WRONG!

Lets take a look at how CenturyLink DSL in Mount Vernon, Ohio routes to Mount Vernon Nazarene University, an OARnet customer,

traceroute to 199.18.160.202 (199.18.160.202), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.554 ms 0.568 ms 0.623 ms
2 65.131.220.254 (65.131.220.254) 8.928 ms 8.947 ms 10.781 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 11.878 ms 13.523 ms 14.449 ms Mansfield, OH
4 cer-edge-12.inet.qwest.net (205.171.139.42) 59.510 ms 59.633 ms 59.652 ms Chicago, IL
5 65.117.112.102 (65.117.112.102) 48.437 ms 48.464 ms 48.463 ms
6 clevs-r5-xe-2-0-0s100.core.oar.net (199.218.39.194) 47.292 ms clevs-r5-xe-2-1-0s100.core.oar.net (199.218.20.25) 48.092 ms 49.180 ms Cleveland, OH
7 clmbn-r5-et-8-0-0s100.core.oar.net (199.218.39.253) 59.893 ms 33.426 ms 34.154 ms Columbus, OH
8 clmbs-r5-et-3-0-0s100.core.oar.net (199.218.20.33) 35.131 ms 35.763 ms 37.584 ms
9 mvnu-r0-vl830.cpe.oar.net (199.18.160.202) 45.064 ms 45.100 ms 45.511 ms Mount Vernon, OH

9 hops to go from Mount Vernon to Mount Vernon, nice… not. I thought CenturyLink peered heavily with OARnet. Well as you can see, we route out to Chicago once again! Giving us a nasty 45ms, when it could be a nice 10ms or below. Tisk tisk..

WOW! (Wide Open West)

WOW! serves Columbus and Cleveland and are well known in Columbus. They are probably one of the best ISPs in Ohio that actually upgrade, offering speeds such as this in the Columbus area,

So does CenturyLink take the courtesy to peer with them in Columbus or Cleveland? Lets find out,

CenturyLink DSL in Mount Vernon, Ohio to WOW! DNS server in Columbus, Ohio,

traceroute to 64.233.222.2 (64.233.222.2), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.638 ms 0.710 ms 0.746 ms
2 65.131.220.254 (65.131.220.254) 9.704 ms 9.631 ms 10.021 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 12.004 ms 12.842 ms 12.976 ms Mansfield, OH
4 chp-brdr-03.inet.qwest.net (67.14.8.194) 25.255 ms 26.279 ms 27.372 ms Chicago, IL
5 63.146.27.18 (63.146.27.18) 28.549 ms 30.189 ms 30.897 ms
6 ae-1-10.bar1.Cincinnati1.Level3.net (4.69.136.213) 52.285 ms 48.683 ms 48.646 ms Cincinnati, OH
7 ae-1-10.bar1.Cincinnati1.Level3.net (4.69.136.213) 48.622 ms 91.278 ms 91.091 ms
8 WIDEOPENWES.bar1.Cincinnati1.Level3.net (4.59.40.18) 26.837 ms 28.847 ms 29.542 ms
9 76-73-167-86.knology.net (76.73.167.86) 32.358 ms 33.031 ms 33.037 ms
10 col11-dns1.col.wideopenwest.com (64.233.222.2) 36.585 ms 36.614 ms 36.889 ms Columbus, OH

Well, still going out to that Chicago border router, ugh! You may argue 36ms is not that bad, but just imagine how much lower this would be if it didn’t go from Ohio > Chicago > Cincinnati > Columbus. Not only that, they don’t even peer in Chicago, going through a tier 1 provider, Level 3. I’m starting to see a pattern, everything outbound goes through that Mansfield aggregation router and then heads to another router in Chicago..

Windstream

Windstream is another ISP that serves Ohio. I’ve mainly only seen them in Newark though. They are a pretty big ISP though.

CenturyLink DSL in Mount Vernon, Ohio to Newark, Ohio Windstream router,

traceroute to 40.129.34.2 (40.129.34.2), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.608 ms 0.750 ms 0.656 ms
2 65.131.220.254 (65.131.220.254) 9.808 ms 9.721 ms 10.617 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 11.421 ms 12.032 ms 13.389 ms
4 chp-brdr-03.inet.qwest.net (67.14.8.194) 24.406 ms 25.260 ms 26.421 ms
5 ae5.cr2-chi1.ip4.gtt.net (173.205.63.229) 28.060 ms 29.216 ms 30.351 ms
6 xe-2-0-0.cr1-chi1.ip4.gtt.net (89.149.187.89) 31.473 ms xe-2-3-0.cr1-chi1.ip4.gtt.net (89.149.185.98) 28.996 ms xe-0-2-7.cr1-chi1.ip4.gtt.net (89.149.187.85) 30.443 ms
7 173.205.35.114 (173.205.35.114) 31.625 ms 119.035 ms 118.890 ms
8 et11-0-0-0.cr01.cley01-oh.us.windstream.net (40.128.248.70) 36.138 ms 37.349 ms 38.500 ms
9 et10-0-0-0.cr02.cley01-oh.us.windstream.net (40.128.248.69) 34.665 ms 36.063 ms et10-0-0-0.agr03.hdsn01-oh.us.windstream.net (40.136.97.134) 32.381 ms
10 et10-0-0-0.agr04.hdsn01-oh.us.windstream.net (40.136.97.136) 38.453 ms 39.378 ms 41.030 ms
11 ge2-2-0-0.pe02.nwrk01-oh.us.windstream.net (40.129.34.2) 46.655 ms 47.775 ms 48.948 ms

Sigh, out to Chicago and back..

AT&T U-Verse

AT&T provides 1Gbps FTTH in quite a few parts of Ohio, while the rest of their Ohio footprint is DSL at a max of 24Mbps. They are a big ISP and cellular provider, you’d think CenturyLink would peer with them in Columbus, but of course, they don’t. Here’s where AT&T offers their 1Gbps FTTH service in Ohio,

It was hard to traceroute to an AT&T IP in Columbus, it seems AT&T’s backbone routers don’t like to respond to ICMP, UDP, or TCP probes. I was able to eventually find a AT&T FTTH Columbus IP to traceroute. Here are the results,

CenturyLink DSL Mount Vernon, Ohio to AT&T FTTH Columbus, Ohio

traceroute to 45.21.103.182 (45.21.103.182), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.360 ms 0.347 ms 0.364 ms
2 65.131.220.254 (65.131.220.254) 10.635 ms 10.863 ms 11.560 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 11.872 ms 13.746 ms 14.858 ms
4 chp-brdr-03.inet.qwest.net (67.14.8.194) 26.137 ms 26.753 ms 28.365 ms
5 63.146.26.218 (63.146.26.218) 33.220 ms 33.376 ms 32.651 ms
6 cr1.cgcil.ip.att.net (12.122.133.106) 42.216 ms 41.898 ms 45.491 ms
7 12.122.161.193 (12.122.161.193) 45.442 ms 30.619 ms 29.018 ms
8 * * *
9 71.158.32.155 (71.158.32.155) 31.867 ms 32.364 ms 33.692 ms
10 71.151.141.183 (71.151.141.183) 37.583 ms 37.614 ms 37.712 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * 71.151.141.183 (71.151.141.183) 28.730 ms !H 38.178 ms !H

We didn’t quite reach our destination, but I did another traceroute from a different server and can verify that our traceroute did hit some AT&T Ohio nodes here. As you can see, we went out to Chicago again, go figure..

So they peer with NONE of the competing ISPs in Ohio, meaning if you want to skype your neighbor on a different IP, you’re gonna have to go out to Chicago and back, thanks CenturyLink. I believe this is mainly a problem due to CenturyLink’s strict peering policies, found here and here .

So lets see if other stuff routes out to Chicago other than competing ISPs in Ohio. Lets see what route it takes to my Linode in Newark, New Jersey. That’s east of Ohio, so it SHOULDN’T have to go west to Chicago at all, that would be inefficient.

traceroute to gtaxl.net (66.175.209.14), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.623 ms 0.499 ms 0.606 ms
2 65.131.220.254 (65.131.220.254) 8.196 ms 9.273 ms 10.688 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 11.231 ms 12.373 ms 13.509 ms
4 chp-brdr-03.inet.qwest.net (67.14.8.194) 24.573 ms 25.657 ms 26.319 ms
5 ae5.cr2-chi1.ip4.gtt.net (173.205.63.229) 27.449 ms 29.329 ms 29.745 ms
6 xe-1-1-2.cr9-nyc3.ip4.gtt.net (89.149.132.81) 42.270 ms 41.097 ms xe-2-1-3.cr9-nyc3.ip4.gtt.net (89.149.181.193) 42.465 ms
7 ip4.gtt.net (173.205.38.198) 44.825 ms 32.251 ms 30.172 ms
8 173.255.239.7 (173.255.239.7) 33.007 ms 33.073 ms 173.255.239.15 (173.255.239.15) 35.703 ms
9 excession.server.gtaxl.net (66.175.209.14) 35.761 ms 39.469 ms 39.515 ms

Chicago it is.. it went west then back east, after it picked up the train in Chicago riding GTT. 😐 However, I notice something interesting, going the other way it uses Zayo and doesn’t go through Chicago. Here’s a trace from my Linode,

traceroute to 65.131.220.254 (65.131.220.254), 30 hops max, 60 byte packets
1 router2-nac.linode.com (207.99.1.14) 0.671 ms 0.828 ms 0.872 ms
2 173.255.239.2 (173.255.239.2) 0.877 ms 0.937 ms 1.010 ms
3 128.177.26.241 (128.177.26.241) 0.928 ms 0.934 ms 0.935 ms
4 ae6.cs2.lga5.us.zip.zayo.com (64.125.30.156) 1.609 ms 1.615 ms 1.616 ms
5 ae27.cr2.lga5.us.zip.zayo.com (64.125.30.253) 1.681 ms 1.683 ms 1.683 ms
6 ae5.er2.lga5.us.zip.zayo.com (64.125.21.81) 2.740 ms 3.520 ms 3.470 ms
7 ae4.er1.lga5.us.zip.zayo.com (64.125.31.165) 1.374 ms 1.473 ms 1.438 ms
8 nyc2-brdr-01.inet.qwest.net (63.146.26.65) 1.437 ms 1.447 ms 1.599 ms
9 * * *
10 65.131.220.254 (65.131.220.254) 22.438 ms 22.543 ms 22.533 ms

Now the 9th hop would definitely be mnfd-agw1.inet.qwest.net, so that means that node has a direct link to nyc2-brdr-01, but on outbound wants to direct me to their Chicago node instead! If on outbound it’d take nyc2 it’d be a MUCH more efficient path!

Amazon Web Services

I was really happy when I heard Amazon will be building a datacenter in Columbus, Ohio. That means I could get an EC2 (basically a VPS) from Amazon in Columbus, Ohio and have nice latency! However, being CenturyLink, they don’t peer with Amazon in Columbus, where Amazon built a datacenter to feed their infrastructure.. Why does CenturyLink hate Columbus so much??!

NIIICE! Lets see what our latency is from Mount Vernon, Ohio.

traceroute to 52.15.170.90 (52.15.170.90), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.646 ms 0.745 ms 0.647 ms
2 65.131.220.254 (65.131.220.254) 8.571 ms 12.713 ms 13.006 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 11.049 ms 12.341 ms 12.954 ms
4 chp-brdr-03.inet.qwest.net (67.14.8.194) 25.029 ms 25.856 ms 26.486 ms
5 ae5.cr2-chi1.ip4.gtt.net (173.205.63.229) 27.628 ms 29.256 ms 29.909 ms
6 ip4.gtt.net (173.205.44.246) 43.256 ms ip4.gtt.net (173.205.44.242) 40.028 ms ip4.gtt.net (173.205.44.246) 40.344 ms
7 52.95.62.54 (52.95.62.54) 71.450 ms 52.95.62.84 (52.95.62.84) 61.656 ms 52.95.62.132 (52.95.62.132) 103.385 ms
8 52.95.62.45 (52.95.62.45) 95.457 ms 97.945 ms 52.95.62.147 (52.95.62.147) 98.013 ms
9 54.239.42.61 (54.239.42.61) 100.480 ms 54.239.42.63 (54.239.42.63) 103.012 ms 103.178 ms
10 54.239.43.213 (54.239.43.213) 106.863 ms 106.886 ms 106.803 ms
11 52.95.2.238 (52.95.2.238) 109.278 ms 109.319 ms 52.95.3.32 (52.95.3.32) 111.821 ms
12 52.95.2.247 (52.95.2.247) 109.148 ms 52.95.2.195 (52.95.2.195) 109.138 ms 52.95.2.113 (52.95.2.113) 54.360 ms
13 52.95.2.222 (52.95.2.222) 54.995 ms 52.95.2.138 (52.95.2.138) 58.262 ms 52.95.2.222 (52.95.2.222) 54.741 ms
14 52.95.2.143 (52.95.2.143) 58.493 ms 52.95.2.115 (52.95.2.115) 55.943 ms 52.95.2.255 (52.95.2.255) 57.550 ms
15 52.95.3.128 (52.95.3.128) 58.588 ms 52.95.1.6 (52.95.1.6) 59.200 ms 52.95.1.2 (52.95.1.2) 59.835 ms
16 * * *
17 * * *
18 * * *

WHY CENTURYLINK, WHY?! :”( The million 52. hops are Amazon’s network, which is a complete mess, but as you can see it doesn’t help CenturyLink routes us out to Chicago first.. So getting a Columbus, Ohio server from Amazon is completely useless to me as a result of CenturyLink’s peering. I’d have MUCH better latency to a Chicago, IL server. :/

CloudFlare

CloudFlare has a HUUUUUGE network and peers with literally everyone they can, and pushes boundaries just to peer with some ISPs.

I was excited to hear they brought a datacenter a little closer to me in Detroit, Michigan. That’s about ~150 miles closer to me than their Chicago, Illinois POP, which is of course where I’ve always routed to. When they brought Detroit online however, I still route to Chicago instead of the closer Detroit. :/ Oh well..

traceroute to cloudflare.com (198.41.214.162), 30 hops max, 60 byte packets
1 pfsense.router.gtaxl.net (10.0.0.1) 0.389 ms 0.452 ms 0.442 ms
2 65.131.220.254 (65.131.220.254) 8.586 ms 8.940 ms 11.295 ms
3 mnfd-agw1.inet.qwest.net (75.160.81.9) 12.323 ms 13.950 ms 14.119 ms
4 chp-brdr-04.inet.qwest.net (67.14.8.238) 37.475 ms 39.098 ms 39.251 ms
5 chi-b21-link.telia.net (213.248.85.29) 41.119 ms 42.527 ms 43.944 ms
6 cloudflare-ic-306330-chi-b21.c.telia.net (62.115.44.10) 44.564 ms 43.530 ms 44.650 ms
7 198.41.214.162 (198.41.214.162) 34.510 ms 18.407 ms 18.327 ms

Since it’s in Chicago, where CenturyLink routes all Ohio traffic, I get decent latency. COULD be better if routed to Detroit, but CenturyLink..

I’ve checked all major DNS resolves, OpenDNS, Google, Level3, etc. and they all route to Chicago. I guess it’s good Google has a POP there so all Google services are speedy, but I’m still annoyed with how CenturyLink treats it’s Ohio customers. I understand Chicago is a huge interconnect on the Internet, so they peer with major Tier 1 providers and ISPs there, but do some peering in Columbus please! Since EVERYONE in Ohio routes out through Chicago, several of their chp-brdr routers (in Chicago) are “on fire” and experience heavy packet loss from time to time. It’s because they put so much load on Chicago for several states like Ohio and Indiana, etc. I had to deal with packet loss today as a result of it. Here you can see the packetloss on that Chicago node of CenturyLink’s. I sent this pastebin to CenturyLink’s support Twitter, with no reply.. Remember, all .qwest IP’s you see is CenturyLink’s backbone. CenturyLink is Embarq + CenturyTel + Qwest. Qwest had the biggest backbone network so they used it.

Anyway, I’m done ranting.. if anyone else in Ohio finds this useful, leave a comment. Hopefully CenturyLink techs will see putting all the load on Chicago is stupid and inefficient.