ISP routing craziness

[go558a83nk 364]

Starting this topic as a way for me to vent and perhaps see that I'm not the only one to see crazy routing by my ISP.

 

The topic has come up in other threads, with knowledgeable people confirming my belief that ISPs relegate VPN traffic to very low priority.

 

Route testing shows that this low priority isn't based on packet inspection but simply the destination of the traffic (and, I assume, the source).  In other words, traffic both directions between us and the server is affected.

 

 

Trace to Draco, primary IP address

2  [7922] [COMCAST-22] xe-3-1-3-sur02.east.tx.houston.comcast.net (68.85.252.81) 8.8ms
 3  [7922] [COMCAST-22] ae-9-ar01.bisbee.tx.houston.comcast.net (68.85.246.65) 7.9ms
 4  [7922] [COMCAST-16] be-33662-cr02.56marietta.ga.ibone.comcast.net (68.86.92.89) 30.7ms
 5  [7922] [COMCAST-16] hu-0-12-0-6-pe01.56marietta.ga.ibone.comcast.net (68.86.89.18) 24.4ms
 6  [7922] [iBONE-CCCS-2] 50.242.148.190 30.9ms
 7  [174] [NET-154-54-0-0] be2847.ccr41.atl01.atlas.cogentco.com (154.54.6.101) 42.4ms
 8  [174] [NET-154-54-0-0] be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70) 28.8ms
 9  [174] [NET-154-54-0-0] be2441.ccr21.dfw01.atlas.cogentco.com (154.54.41.66) 30.1ms
10  [174] [NET-154-54-0-0] te0-0-0-0.agr12.dfw01.atlas.cogentco.com (154.54.31.114) 41.1ms
11  [174] [NET-154-24-0-0] te0-0-2-3.nr11.b000821-1.dfw01.atlas.cogentco.com (154.24.19.222) 37.1ms
12  [174] [COGENT-A] 38.88.50.34 35.6ms
13  [15003] [NETBLK-NOBIS-TECHNOLOGY-GROUP-06] [target] 64.120.63.90 32.1ms
 

Note it goes all the way to Atlanta, then back to Dallas through Houston on Cogent.  Why on earth do they send it to Atlanta to get on Cogent's network?  Dallas is only 4 hours drive from Houston and as you'll see from the next trace, Comcast has a connection to Cogent right in Houston.

 

Trace to Draco, alternate IP address

2  [7922] [COMCAST-22] xe-3-1-2-sur03.east.tx.houston.comcast.net (68.85.251.245) 7.7ms
 3  [7922] [COMCAST-22] ae-18-ar01.bearcreek.tx.houston.comcast.net (68.85.246.69) 8.6ms
 4  [7922] [COMCAST-16] be-33662-cr02.dallas.tx.ibone.comcast.net (68.86.92.61) 15.6ms
 5  [7922] [COMCAST-16] be-12493-pe01.houston.tx.ibone.comcast.net (68.86.84.158) 20.4ms
 6  [7922] [CBC-COMCAST-1] 173.167.59.42 20.5ms
 7  [174] [NET-154-24-0-0] te0-0-1-0.rcr12.iah02.atlas.cogentco.com (154.24.26.89) 21.0ms
 8  [174] [NET-154-54-0-0] be2145.ccr41.iah01.atlas.cogentco.com (154.54.1.85) 24.1ms
 9  [174] [NET-154-54-0-0] be2441.ccr21.dfw01.atlas.cogentco.com (154.54.41.66) 20.1ms
10  [174] [COGENT-NB-0000] te0-0-0-0.agr11.dfw01.atlas.cogentco.com (66.28.4.50) 21.0ms
11  [174] [NET-154-24-0-0] te0-0-2-0.nr11.b000821-1.dfw01.atlas.cogentco.com (154.24.15.50) 21.7ms
12  [174] [COGENT-A] 38.88.50.34 24.2ms
13  [15003] [NETBLK-NOBIS-TECHNOLOGY-GROUP-06] [target] 64.120.63.92 20.4ms
 

Amazingly, the trace gets to Dallas in the 4th hop, but then heads right back to Houston to get on Cogent's network.  It makes zero sense unless you realize that Comcast are doing this crap on purpose.  I have rarely seen a route that heads directly to the VPN server upon reaching Dallas, finding Cogent's network there instead of having to get back to Houston.

 

Understand that this is normal, everyday behavior.  I have another VPN provider with servers in Dallas and Atlanta.  Amazingly, all the Atlanta servers route through Dallas, and all the Dallas servers route through Atlanta.  So, the routes are available, but purposely screwed up.

 

Finally, check this out.

 

Instead of tracing the route to Draco, I instead trace the route to Cogent's router at 38.88.50.34, seen in hop 12 above.  Check out the premium routing I get now.

 

 2  [7922] [COMCAST-22] xe-3-1-3-sur03.east.tx.houston.comcast.net (68.85.251.249) 10.2ms
 3  [7922] [COMCAST-22] ae-18-ar01.bearcreek.tx.houston.comcast.net (68.85.246.69) 9.1ms
 4  [7922] [COMCAST-16] be-33662-cr02.dallas.tx.ibone.comcast.net (68.86.92.61) 14.1ms
 5  [7922] [COMCAST-16] be-12495-pe03.1950stemmons.tx.ibone.comcast.net (68.86.85.194) 13.7ms
 6  [7922] [iBONE-CCCS-3] 50.248.118.246 13.0ms
 7  [174] [NET-154-54-0-0] be2763.ccr21.dfw01.atlas.cogentco.com (154.54.28.73) 14.1ms
 8  [174] [NET-154-54-0-0] te0-0-0-0.agr12.dfw01.atlas.cogentco.com (154.54.31.114) 13.8ms
 9  [174] [NET-154-24-0-0] te0-0-2-3.nr11.b000821-1.dfw01.atlas.cogentco.com (154.24.19.222) 15.2ms
10  [174] [COGENT-A] [target] 38.88.50.34 13.6ms

 

[OpenSourcerer 1441]

Do you want others to post their results or are you specifically invite Comcast customers to do so?

[go558a83nk 364]

Do you want others to post their results or are you specifically invite Comcast customers to do so?

 

if you've got crazy routing too then you're most welcome to post!  just don't make me jealous by posting awesome routing.

[larky 40]

The (partial) answer to your question lies with this statement you made: "Why on earth do they send it to Atlanta to get on Cogent's network? Dallas is only 4 hours drive from Houston and as you'll see from the next trace, Comcast has a connection to Cogent right in Houston"
 
Your statement (bold red part) assumes that routing is based upon geological constraints, it isn't, Comcast may have a connection to "Cogent in Houston" but the server cluster handling the route may be on a different path rather just a simple straight concept connection to "Cogent in Houston". Its a little confusing but i'll try to briefly explain .. so basically:

 

For example, the first server you hit in the Comcast network is semi-random depending upon loading, traffic priority, and the data center assignment it hits, and routing priority, then what happens from there is; next server you hit in Comcast network determines the route priority and path through the Comcast network. Notice in your first trace (Trace to Draco, primary IP address) this > "xe-3-1-3-sur02.east.tx.houston.comcast.net (68.85.252.81)" then goes to "ae-9-ar01.bisbee.tx.houston.comcast.net (68.85.246.65)" - next - notice in the second trace (Trace to Draco, alternate IP address) >  "xe-3-1-2-sur03.east.tx.houston.comcast.net (68.85.251.245)" then goes to "ae-18-ar01.bearcreek.tx.houston.comcast.net (68.85.246.69)" and from the second server (# 3 in each of your traces) it goes into the Comcast internet backbone. Notice that the first server you hit in your traces all have a different IP address but the name is similar (e.g. ...sur03.east.tx.houston.comcast.net ) in your first trace you hit server group "xe-3-1-3" ("xe-3-1-3-sur02.east.tx.houston.comcast.net" and a server in that group with the IP address 68.85.252.81) in the second trace you hit a different server group but in the third trace you hit the same server group as the first trace but a different server in that group (ip address 68.85.251.249). Geological aspects do not matter in the routing through the Comcast network. It just so happens that in the Comcast network there can be a server group but the group is split into IP address groups in different Comcast centers which may actually be geologically separated e.g. server group xe-3-1-3 may have part of their servers in Philadelphia and part of them in Texas but they are all the same server group even though the name is ..east.tx.houston.comcast.... notice the difference in the names of sur02 and sur03 indicating they are actually in different sub groups which in the Comcast network usually means they are in different geological locations.

 

In your next comment: "So, the routes are available, but purposely screwed up" - nope, not really, its normal activity per design of the network. As you discovered with what you think is "premium routing" is that the end point you are trying to reach also plays a part in the routing - you changed the routing priority by tracing a route to "Cogent's router at 38.88.50.34" as end points in the Comcast network or data center get higher priority so they get the shortest route most times, depending on where the server cluster is located network wise (not geographically or rack placement wise) based upon the initial route chosen from the Comcast network which may appear geologically related but it isn't actually its just that it happens to be in the same data center cluster.

 

Are you using Comcast business or residential service?

[go558a83nk 364]

The (partial) answer to your question lies with this statement you made: "Why on earth do they send it to Atlanta to get on Cogent's network? Dallas is only 4 hours drive from Houston and as you'll see from the next trace, Comcast has a connection to Cogent right in Houston"

 

Your statement (bold red part) assumes that routing is based upon geological constraints, it isn't. Its a little confusing but i'll try to briefly explain .. so basically:

 

For example, the first server you hit in the Comcast network is semi-random depending upon loading, traffic priority, and the data center assignment it hits, and routing priority, then what happens from there is; next server you hit in Comcast network determines the route priority and path through the Comcast network. Notice in your first trace (Trace to Draco, primary IP address) this > "xe-3-1-3-sur02.east.tx.houston.comcast.net (68.85.252.81)" then goes to "ae-9-ar01.bisbee.tx.houston.comcast.net (68.85.246.65)" - next - notice in the second trace (Trace to Draco, alternate IP address) >  "xe-3-1-2-sur03.east.tx.houston.comcast.net (68.85.251.245)" then goes to "ae-18-ar01.bearcreek.tx.houston.comcast.net (68.85.246.69)" and from the second server (# 3 in each of your traces) it goes into the Comcast internet backbone. Notice that the first server you hit in your traces all have a different IP address but the name is similar (e.g. ...sur03.east.tx.houston.comcast.net ) in your first trace you hit server group "xe-3-1-3" ("xe-3-1-3-sur02.east.tx.houston.comcast.net" and a server in that group with the IP address 68.85.252.81) in the second trace you hit a different server group but in the third trace you hit the same server group as the first trace but a different server in that group (ip address 68.85.251.249). Geological aspects do not matter in the routing through the Comcast network.

 

In your next comment: "So, the routes are available, but purposely screwed up" - nope, not really, its normal activity per design of the network. As you discovered with what you think is "premium routing" is that the end point you are trying to reach also plays a part in the routing - you changed the routing priority by tracing a route to "Cogent's router at 38.88.50.34" as end points in the Comcast network or data center get higher priority so they get the shortest route (most times, depending on where the server cluster is located based upon the initial route chosen from the Comcast network) which may appear geologically related but it isn't actually its just that it happens to be in the same data center cluster.

 

Are you using Comcast business or residential service?

 

Residential service.  Business is outrageously priced (more than double) for the same speed.

 

My opinion on the matter is that route quality/priority based on end point should be considered anti net neutrality. 

 

Also, there's nothing random about it.  Routing may change from time to time but not randomly or depending on time of day (load).

[larky 40]

Sure it changes and is semi-random, you just don't know it and you can't discover it via route tracing from your end either as it takes special testing and procedures and you have to be at a Comcast server center to test it. Read my edited post above. Its been one of our biggest headaches in testing the Comcast network for compliance. There is a lot more to your routes than you see, for example, I can tell you that between your #2 and #3 servers in your trace you hit an additional two servers in the Comcast network that will not show up on a route test from your end and your route after your #2 went from Texas to Philadelphia to Atlanta then back to Texas before your #3 server. All Comcast traffic will pass through the Philadelphia - Atlanta route after the first hop (your #2) and before the second hop (your #3) and it will never show up on a route test from your end. To make it more confusing, in the Comcast network the IP address of the server you hit in the server group is not the actual server that's doing the routing, its a front end server and behind that is another server group that actually doing the work with routing and you never see these either from your end in route testing. 

[go558a83nk 364]

Sure it changes and is semi-random, you just don't know it and you can't discover it via route tracing either as it takes special testing and procedures and you have to be at a Comcast server center to test it. Read my edited post above. Its been one of our biggest headaches in testing the Comcast network for compliance.

 

I'm not sure what you edited but I'll take your word for it.  Also, re-read my line in the OP about the other VPN service I use and the routing to their servers in Dallas and Atlanta.  There's nothing random about it.  It's a purposeful screwing up of routes.

 

I do want to hit on the last paragraph more.

 

You imply by saying "end points in the Comcast network or data center get high priority" that the cogent router 38.88.50.34 is in Comcast's network or datacenter and that's why that trace has better routing.  In fact it's 4 hops out of comcast's network.  Can you explain what you meant by that?

[larky 40]

Ok, believe what you want. Its a complicated subject when it comes to Comcast, and i'm not going to spend time arguing semantics and personal beliefs. I tried to help with personal intimate knowledge from my professional life. I'm out.

[go558a83nk 364]

Ok, believe what you want. Its a complicated subject when it comes to Comcast, and i'm not going to spend time arguing semantics and personal beliefs. I tried to help with personal intimate knowledge from my professional life. I'm out.

 

That isn't what I want at all.  Just because I'm emotionally hard-headed about it doesn't mean I intend to argue with you and run you off.  I appreciate very much the time you took to talk about it with me.

[John Gow 18]

On 4/3/2017 at 9:33 AM, go558a83nk said:

 

Residential service.  Business is outrageously priced (more than double) for the same speed.

 

My opinion on the matter is that route quality/priority based on end point should be considered anti net neutrality. 

 

Also, there's nothing random about it.  Routing may change from time to time but not randomly or depending on time of day (load).

One of the founding principles of the internet, rooted in its original mission as a purely military net for US defense purposes, was that traffic should not be centralized or based upon proximity.  This was important because they wanted a network that was adaptive, that couldn't be easily sabotaged by the Soviets.  If the national network were linear, you could take out a few hubs and kaboom, we're in the dark.
This is also, I believe, the only reason why VPNs "work." We needed a way to poke holes in the hubs, should the centers of communication be compromised. Ironically (perhaps not) the NSA funds TOR and many of TOR nodes are operated by them... purely for the good of us all, of course.  Please correct me if my understanding is incorrect.