Am I crazy, or is my ISP throttling OpenVPN?

[rainmakerraw 94]

I'm with Virgin Media in the UK, on 160/12 cable.

 

Last year I had a spate of low speed (3MB/sec hard cap) which I initially blamed on throttling of OpenVPN as I could hit full speed on my naked ISP connection. After some investigation I found it was actually a bug in the ISP supplied router, so I switched to my own and the problem went away. 

 

Lately however, I'm having a hard speed cap problem and it really looks like issues caused by either VM's use of DPI and/or OpenVPN throttling/shaping at ISP level. VM operate a whitelist for shaping, so unless the protocol is whitelisted it's shaped by default. VM categorically and publicly deny any form of throttling, shaping or interference with OpenVPN connections. 

 

I've been using an Ubuntu torrent as a speed benchmark as it's multi-threaded, consistently very fast, and can be used off-VPN without fear of legal issues. I have tested every port and protocol in Eddie, as well as via Viscosity (to rule out Eddie issues). I also tried the same tests with several other well respected VPN providers with good networks and the results were consistent across them all, Air included. 

 

Note that I am using MB/sec in its proper format, meaning megabytes per second. 1MB/sec = 8Mbps. All results are for the same Ubuntu 15.04 x64 torrent downloaded in the latest qBittorrent v3.2.3 on Mac OS X (also verified on Linux, PCBSD and Windows 8.1 Pro). As well as checking against multiple VPN companies, multiple OpenVPN software and multiple operating systems, I also reproduced the results on multiple machines (mid 2012 MacBook Pro and my FX8350 / 16GB DDR3 / Samsung Evo 850 sad / Radeon R9 380 gfx desktop). 

 

I repeated the tests with several ethernet cables (to rule out cable issues), as well as with *machine* > router > modem and *machine* > modem (to rule out firmware or routing issues). Every time, regardless of the variable, the results below were consistent.

 

ISP : 19MB/sec

 

OpenVPN 53 UDP : 2MB/sec

 

OpenVPN (all other ports in turn) UDP : 5MB/sec

 

OpenVPN (all ports) TCP : 4 - 5 MB/sec

 

OpenVPN + SSH 22  : 2MB/sec

 

OpenVPN + SSH 80 (or 53) : 13 - 18 MB/sec (lower in peak times, high off-peak)

 

OpenVPN + SSL 443 : 13 - 18 MB/sec (lower in peak times, high off-peak)

 

As we can see, generally SSL and SSH masking the OpenVPN connection allows almost full line speed (minus the encryption overheads). That's great. As soon as it's a bare OpenVPN connection the speeds cap out at around 33% of what they should be. Bare OpenVPN TCP is a little slower than UDP (as you'd expect) but otherwise in accordance with the general 5MB/sec cap experienced on UDP. The only exceptions are UDP:53 and SSH:22 which are both heavily restricted to around 2MB/sec. 

 

Now to my mind, knowing what I do of VM's shaping and DPI systems, this would only make sense if they were interfering with OpenVPN either by purposefully throttling it, or else their DPI system is messing up the connection. They further seem to restrict SSH:22 and UDP:53 by protocol but not by port. This actually makes sense, as all other Eddie combinations are quite random whereas SSH:22 (SSH) and UDP:53 (DNS) are established network traffic protocols and thus could be singled out for listing in the shaping systems. If we reverse the protocol/port (to give SSH 53 and UDP 22) we once again obfuscate the tunnel and go back to full speeds! 

 

I also get a lot of decrypt/replay errors in the logs on every single port for 'normal' OpenVPN. As soon as I hide the OpenVPN in either SSL or SSH the errors simply don't occur. Ever. This suggests that the extra tunnel is hiding the OpenVPN tunnel from being shaped, or else the DPI process in and of itself is breaking OpenVPN and causing the packets to arrive out of order. Maybe that in and of itself can hurt speed? 

 

So there you go. Sorry for the long post but it's an interesting (if thoroughly frustrating and annoying) issue. What do you gurus think? Given I have worked to change the variables one at a time to rule out issues with AirVPN (different providers), the router and/or its firmware (direct connection to modem, bypassing router), wireless issues (used ethernet directly) and OS limits or bugs (used multiple OSs) I can't see anything is left... except issues with the ISP shaping/throttling or else their DPI breaking things. 

 

I posted a thread very similar to this in VM's support forums, but for a whole week it has gone unanswered by any staff. Interestingly it is the only thread on the forum to have been ignored. Make of that what you wish.

 

I await your replies with interest. Thanks in advance for reading.

[zhang888 1066]

Looks like an interesting experiment indeed, however, it seems like you forgot to exclude a vital parameter in this mini-scientific research,

which is - your ISP at any time of the test cases, as an excluding problematic vector of the control group

 

These tests would have been much more conclusive if you could try TalkTalk, Sky, BT.

 

The vast majority of residential ISPs throttle traffic, and that's somewhat acceptable. Most users can't find the difference in any case,

while the heavy users are simply not very profitable to keep on those advertized speeds.

I don't know how much you pay for that 160/12 residential line, but the traffic you could consume while utilizing it 24/7 should be about

20TB/month. This is very unprofitable for a home ISP to do, just to compare, I would have to pay about 100 GBP for such volume in the UK.

So keeping users at the minimum seem to be a common practice.

 

Now some ISPs take the "eat all you can" approach, knowing that 99,9% of the users won't exceed the 20TB/mo theoretical limit, and thus

do not throttle connections that much. Sort of like Air is doing, they face the same "issue" with the heavy users.

[rainmakerraw 94]

Ideally of course I could test on a different ISP, but everyone I know is on VM's cable. Since max throughput on the other main UK ISP's isn't much more than the 'cap' I'm experiencing it also wouldn't tell me too much.

 

Your hypothesis about the ISP limiting data throughput to maintain profits would make sense except for two factors (which are related):

 

1) Virgin Media have recently removed all downstream limits (peak time traffic based speed restrictions)

2) I can get full speed instantly by switching off the VPN. 

 

If they were doing it to save money or bandwidth, surely it would apply to the whole connection and not just to OpenVPN connections on the line. BTW I would gladly pay £100 a month for the service to work properly. Seriously. If I could get 160/12 (or better yet symmetric gigabit like Gigaclear or B4rn) that worked at rated speeds consistently I would gladly pay that. It's only £25 a week for something I use heavily. Cheaper than smoking anyway and a lot more fun haha.

 

The irony is companies like Gigaclear provide symmetric gigabit (and 10Gbps) for about £70 a month and are widely reported to be un-contended, superbly reliable and protocol agnostic. Shame they are only in a few places.

[bluesjunior 43]

I am certainly no guru but I do also use VM via their box in modem mode only and then out to a Buffalo WHR-G300N router upgraded with DD-WRT. I am just using the Windows 7 connection for AirVPN as the router doesn't have the open VPN option. I have been using AirVPN around a year now and like you have noticed differences in speed. A year ago I just used the best server available option in AirVPN which was usually Dutch but got access problems with a couple of my favourite forums so did some speed checks with the UK servers and found that I got the best speed/bandwidth from Naos and started using it with no internet blocking problem. Using the Speedtest website I would get around 50% of my 120mbps bandwidth through Naos and the full whack with AirVPN off. I also found throttling issues with my specific port forwarded qBittorrent client only getting a max of 2mbps on heavily seeded ones but usually much less. At that time I read some posts here and tried different protocol settings settling eventually on SSL Tunnel port 443 where I got an average of 4mbps on my bittorrents and used this set up until last week when Naos was down for servicing. I went back to connecting through the best available server again and again it was the Dutch ones although not the same ones as before. Using the Speedtest site again I found that I was getting 100-110 of my 120mbps connection and after reading another post here I changed to Protocol UDP Port 443 and found that I could now get 7mbps download speeds on my qBittorrent client which was the best I had seen even prior to using a VPN the most I got was up to 4mbps. A thing I noticed since changing protocols from the tunnel to UDP is that I sometimes now get thousands of network intrusion attemps in my Comodo CIS. Not sure if anything here helps but I will also be watching the replies to this thread as we have a common interest in this.

[zhang888 1066]

I will add (fair disclosure: I used to integrate DPI solutions in various ISPs in the past) that most of them - Allot, Sandvine and BlueCoat use connection pp/s based limiting, together

with popular content caching.

So that when anyone downloads a new popular content over HTTP, this will most likely be served from a local cached storage. This makes the popular Linux .iso's tests kind of useless.

 

Regarding torrents, what happens is quite natural - you don't hit any pp/s rate limit with a regular download - assuming you have about 200 peers in the swarm, from which 20 of them

are seeders, you will probably get less than 1MBps download speed from each one of them. Not enough to hit the thresholds:

 

 

But, when you connect to VPN, from the ISP's DPI side it looks like you are trying to maximize traffic on a single UDP/TCP session, and that is something those devices don't like.

This is a very basic thing that I believe even set to be on by default, also to prevent bunch of other things like outgoing floods.

 

The only solution is to use a known service which is known to consume bandwidth on the same tuple, which leaves us to SSL over 443, SSH over 22, probably NNTP over 119 and a few more

that will be less throttled with default ISP settings. But this also varies among ISPs and their DPI equipment of their choice. Back in 2010 there were only 3 of those DPI companies with some

basic rulesets, nowadays this number is for sure much higher and the algorithms are more sophisticated. Especially for a country like UK.

[rainmakerraw 94]

Interesting post, thanks! The ~5MB/sec download 'limit' has only recently affected me; until now I've always enjoyed full speed regardless of protocol and encryption (or lack thereof). Maybe they 'upgraded' their systems? Who knows. So long as SSL and SSH keep on serving the goods I suppose it's just a minor annoyance, more for the fact of feeling restricted in my choices than any actual inconvenience.

 

It does restrict one's OpenVPN providers rather significantly, however. Luckily Air is a good egg, but things change. If Air ever became unsatisfactory I'd be severely limited in my options. That's an uncomfortable feeling as 99% of VPN providers don't allow connections via SSH or stunnel, and the few that do are often rubbish in other ways. I used to be able to max my line on any VPN provider. Now I'm basically limited to one. Cheers.

[Ernst89 11]

Firstly zhang888's point that multiple connections achieve greater throughput due to limiting per connection may be true but the same effect is observed due to nothing more than normal congestion. On a congested Virgin Media line many connections through a tunnel will give lower throughput than many separate connections outside the tunnel.

 

Your arguments are generally convincing but getting a good speed OpenVPN connection is often hard work and so it is very hard to exclude alternative hypotheses to shaping. I also suspect that you may have over egged the pudding claiming to have tested on multiple VPS providers, the test only really works if you can flip between SSL hidden OpenVPN and normal OpenVPN and even then the SSL and non SSL services may be different.

 

I had a quick look at a product called Glasnost which claims to detect P2P shaping but it didn't seem to do OpenVPN. Maybe someone at AirVPN could develop a similar product to detect OpenVPN shaping?

 

For what it is worth my Virgin Media connection does not appear to be shaped, in that I still get full speed, Obviously that doesn't mean you aren't being shaped. Virgin Media provides notoriously inconsistent service levels to different customers.

 

Virgin Media do state that they use dpi to shape P2P and Usenet but they don't explicitly mention OpenVPN or any VPN. I would try starting a new thread on the Virgin Media forums repeating your previous post and stating that it was ignored by staff. I suspect the Virgin Media staff on the Forum do not know the answer and hence don't want to reply.

[rainmakerraw 94]

Firstly zhang888's point that multiple connections achieve greater throughput due to limiting per connection may be true but the same effect is observed due to nothing more than normal congestion. On a congested Virgin Media line many connections through a tunnel will give lower throughput than many separate connections outside the tunnel.

 

Your arguments are generally convincing but getting a good speed OpenVPN connection is often hard work and so it is very hard to exclude alternative hypotheses to shaping. I also suspect that you may have over egged the pudding claiming to have tested on multiple VPS providers, the test only really works if you can flip between SSL hidden OpenVPN and normal OpenVPN and even then the SSL and non SSL services may be different.

 

I had a quick look at a product called Glasnost which claims to detect P2P shaping but it didn't seem to do OpenVPN. Maybe someone at AirVPN could develop a similar product to detect OpenVPN shaping?

 

For what it is worth my Virgin Media connection does not appear to be shaped, in that I still get full speed, Obviously that doesn't mean you aren't being shaped. Virgin Media provides notoriously inconsistent service levels to different customers.

 

Virgin Media do state that they use dpi to shape P2P and Usenet but they don't explicitly mention OpenVPN or any VPN. I would try starting a new thread on the Virgin Media forums repeating your previous post and stating that it was ignored by staff. I suspect the Virgin Media staff on the Forum do not know the answer and hence don't want to reply.

 

At first glance your rationale about (multi-threaded) out of tunnel speeds being better than (single threaded) in-tunnel speeds made sense... Then I remembered that the low speed only happened over 'naked' OpenVPN whereas OpenVPN inside SSH 53 or OpenVPN inside SSL 443 (similarly single threaded, as it were) reach practically wire speed. You're right about the providers in a sense. I know Air don't like competitor advertising on the forum, but since these results are not an endorsement but rather raw data (and in any case make Air look like the only decent provider lol) I think it should be OK:

 

Tunnelr (BSD based provider) OpenVPN 443 UDP : 5 MB/sec 

Tunnelr SSH tunnel : 3MB/sec (they weren't good as a company so not taking this too seriously)

 

Proxy.sh OpenVPN 443 UDP : ~4MB/sec or so

Proxy.sh OpenVPN over stunnel : 10MB/sec

 

PIA OpenVPN 8080 UDP : 6MB/sec tops but usually 4 - 5 MB/sec [Note PIA don't allow UDP 443)

 

And it goes on and on (VPN.ac, TigerVPN, CactusVPN, BolehVPN, PrivateVPN, NordVPN etc etc etc...). All similar results but none come close to air for the overall package, port forwarding and DNS services, speed and consistency, client or customer service. The point being, though, that they all pretty much demonstrate a much better speed when the OpenVPN connection is further obfuscated inside another tunnel. Just this second on my desktop rig I installed an Intel Pro 1000GT NIC and ran Eddie OpenVPN:443 over SSL = 14MB/sec. I hit disconnect, went into preferences and changed to bare UDP:443 and double clicked the same server to reconnect. Total time taken, less than 10 seconds. Speed now = 5MB/sec. Far too coincidental to always hit that same number time and again. I would have blamed something LAN side but for the fact I have tried several machines and OSs directly wired (cat5e and cat6 tested) to the modem to rule out other issues.

 

It could well be congestion playing its part. We're on a new build estate and new streets are being added regularly. We have had four streets appear and become full of people in literally the last month alone. One can only imagine this is leading to ever more congestion on the local infrastructure. I suppose if VM keep on ignoring me we'll never really know. I really wanna move 30 mins down the road and get symmetric gigabit FTTP for £69 a month... My wife is less keen lol.