speedtest comparison

[dr_kristau 6]

Hello,

I've attached a couple of images containing the results from speedtest.net for two tests - one using my ISP (915Mbps max) and the other using AirVPN (82Mbps). As we can see, using an AirVPN server about 300 miles (about 480km) from my location means - at least according to speedtest.net - taking a speed drop by a magnitude of over ten.

I was wondering what the AirVPN community had to see about such results. Are they of the magnitude expected? How reliable does the community think there are? Many thanks for any suggestions.

[Tech Jedi Alex 1499]

You're on fiber, right? Because if so, you are not the first with this, and you won't be the last. I can't wrap my head around it myself because I don't know anyone who is on fiber to test anything (I'm in Germany, after all), but all the people before you suggest that OpenVPN is problematic with fiber connections.

[dr_kristau 6]

Thank you for your reply!

Yes, I'm on FTTP - I would be very interested in hearing some technical explanations of the problems OpenVPN has with fiber, and what (if anything) I can do to mitigate them.

Thanks again.

 

[Tech Jedi Alex 1499]

1 minute ago, dr_kristau said:

I would be very interested in hearing some technical explanations of the problems OpenVPN has with fiber, and what (if anything) I can do to mitigate them.

Everyone is, including me. ¯\_ (ツ)_/¯

[dr_kristau 6]

9 minutes ago, giganerd said:

Everyone is, including me. ¯\_ (ツ)_/¯

Well, thanks for letting me know that it is OpenVPN which is the problem - searching around the place I've found this interesting looking document:

https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

I'll try some of these tweaks to see if they make any difference - any more suggestions or insights will be gratefully received, however!
 

[Tech Jedi Alex 1499]

While I take note of what is written there, the fact it works with a 250 Mbit/s DSL connection without a fuss makes me wonder a bit. It also worked with a 400 Mbit/s DOCSIS 3.0 connection in the past. Both on Linux, both default parameters; I don't even want to start talking about Windows if simply replacing the tunnel driver doubles the throughput..

[dr_kristau 6]

20 minutes ago, giganerd said:

While I take note of what is written there, the fact it works with a 250 Mbit/s DSL connection without a fuss makes me wonder a bit. It also worked with a 400 Mbit/s DOCSIS 3.0 connection in the past. Both on Linux, both default parameters; I don't even want to start talking about Windows if simply replacing the tunnel driver doubles the throughput..

Windoze lol.

I run OpenVPN on a little barebones machine with an Ubuntu server which acts as a router. If I can't get significant improvements with these tweaks, then I might try AirVPN's Hummingbird again - the last time I tried it I couldn't get it to work properly on my 'router', and suspected that it was designed as a client for individual workstations.

[dr_kristau 6]

1 hour ago, dr_kristau said:

If I can't get significant improvements with these tweaks

I've tried adding the following to the end of my AirVPN-created ovpn file:
 

proto udp
tun-mtu 48000
txqueuelen 1000
fragment 0
mssfix 0
engine rdrand
auth SHA512
<ca>
-----BEGIN CERTIFICATE-----

and it has made no difference to the speed at all, that I can detect.

[dr_kristau 6]

5 hours ago, dr_kristau said:

If I can't get significant improvements with these tweaks, then I might try AirVPN's Hummingbird again

I've given Hummingbird a go, and this time I was able to use it successfully - it created a TUN interface seamlessly and my 'router' worked as expected. Interestingly, before it worked, it made the following complaints about the parameters I had changed in my AirVPN-created ovpn file:
 

Error: eval config error: ERR_PROFILE_OPTION: option_error: mssfix: parse/range issue

 

OpenVPN3 CONNECT ERROR: option_error: sorry, 'fragment' directive is not supported, nor is connecting to a server that uses 'fragment' directive

 

TUN Error: tun_prop_error: tun_builder_set_mtu failed

taking these three parameters out of the ovpn file (mssfix, fragment, and tun-mtu) enabled Hummingbird to run correctly. However, I was greatly disappointed with the speed performance - running through Hummingbird gave me speeds which were less than half of those I obtained using the v.2.4.7 of OpenVPN, as you can see in the attached files. Maximum speeds for AirVPN dropped from 82Mbps to 37Mbps (907Mbps with AirVPN unplugged for comparison).

 

[Tech Jedi Alex 1499]

Hummingbird is based on OpenVPN 3 which did not reach feature parity with OpenVPN 2 yet, so some options are ignored.
Your observation regarding Hummingbird performance further suggests that there is a problem with OpenVPN over fiber, but I can't imagine at all what it could be.

[Staff 10334]

Hello!

On a 1 Gbit/s fiber line in Italy which does not suffer traffic shaping we record about 330 Mbit/s with Hummingbird and an Intel i7 (in a Fedora 32 system), with AES-256-GCM on Data Channel and connections to Belgium and the Netherlands servers. Hummingbird uses OpenVPN3 linked against mbedTLS, but we record same performance with OpenVPN 2 linked against OpenSSL.

Similar, consistent speeds are recorded by many users. Consider that we have in particular a couple of customers who connect only from dedicated servers and keep their speed (with OpenVPN 2 or 3) constantly at 480 Mbit/s (i.e. 960 Mbit/s on the server).

What are your system, CPU, OS? Are you sure that your ISP does not enforce traffic shaping and that your CPU is not the bottleneck? Have you tested several servers to maximize likelihood of good peering between your ISP and our transit providers?

Kind regards
 

[dr_kristau 6]

12 hours ago, Staff said:

What are your system, CPU, OS? Are you sure that your ISP does not enforce traffic shaping and that your CPU is not the bottleneck? Have you tested several servers to maximize likelihood of good peering between your ISP and our transit providers?

I've just finished looking at that - I ran an experiment, using the CLI version of speedtest.net, and identical versions of OpenVPN/Hummingbird and my *.ovpn AirVPN configuration file. I was surprised to find that the results I detailed above I can reproduce running directly from my barebones 'router' (which is running Intel Celeron J1900), but when I run the same on my desktop (which is running Intel Core i7-4771) the results over the VPN are a magnitude of 3-4 times faster. Is OpenVPN/Hummingbird performance really so dependent on CPU capacity, do you think, or are there other things I should be looking at? Is there any way I can tweak the J1900 configuration? Both my 'router' and my desktop are running Ubuntu 20.04.1, the 'router' is running the server version while the desktop is running the desktop version.

[dr_kristau 6]

12 hours ago, dr_kristau said:

Is OpenVPN/Hummingbird performance really so dependent on CPU capacity, do you think, or are there other things I should be looking at?

I'm coming to the conclusion that OpenVPN (both 2 and 3) are extremely CPU resource-intensive, which I believe was one of the motivations for the productions of new VPN protocols such as WireGuard. I'm now wondering how much better a WireGuard client would perform on the limited CPU resources on my barebones 'router'. One way to test this would be to contract a droplet on DigitalOcean, say, deploy something like algo on it, and give it a try. At least this way I'd avoid the questionable privacy aspects of WireGuard by installing it on a server to which only I have access.  

[Tech Jedi Alex 1499]

I heard it can handle a gigabit connection.

[Staff 10334]

@dr_kristau

Hello!

Yes, AES-256-GCM is computationally hard for non AES-New Instructions supporting CPUs. Also consider that OpenVPN 2 does not scale and runs in a single thread of a single core, in the userspace. You have a very slight performance improvement when OpenVPN is linked against mbedTLS and not OpenSSL, but it's not really essential..

With your Celeron you should get significant performance improvement with CHACHA20-POLY1305 cipher for Data Channel. For that, you will need OpenVPN 2.5 or higher version (OpenVPN 2.5 stable version was released yesterday).

We have completed deployment of OpenVPN 2.5 on all servers now and while we restart the daemons, more and more VPN servers will offer CHACHA20-POLY1305 on both Data and Control Channel. At the moment, you can have CHACHA20-POLY1305 on the servers marked as "Experimental". Remember that OpenVPN 2.5 or higher version is required, as older versions do not support CHACHA20 on Data Channel.

Kind regards
 

[dr_kristau 6]

4 hours ago, Staff said:

@dr_kristau
Remember that OpenVPN 2.5 or higher version is required, as older versions do not support CHACHA20 on Data Channel

Thank you.

I followed the instructions here and have installed OpenVPN 2.5. I then generated a new AirVPN config file called AirVPN_NL-Alblasserdam_Comae_UDP-443.ovpn, which I modified as follows:

push-peer-info
setenv UV_IPV6 yes
remote-cert-tls server
cipher CHACHA20-POLY1305
comp-lzo no
proto udp
key-direction 1
<ca>
-----BEGIN CERTIFICATE-----

Starting from my FTTP baseline:

    Latency:     2.70 ms   (0.18 ms jitter)
   Download:   923.12 Mbps (data used: 680.1 MB)                               
     Upload:   290.74 Mbps (data used: 250.2 MB)                               
Packet Loss:     0.0%

I ran this modified Comae config file on my desktop i7-4771 CPU and obtained:

    Latency:    67.46 ms   (0.33 ms jitter)
   Download:   310.91 Mbps (data used: 268.9 MB)                               
     Upload:    65.17 Mbps (data used: 78.8 MB)                               
Packet Loss:     1.2%

I then ran it again on my 'router' J1900 CPU and obtained:

    Latency:    67.90 ms   (0.25 ms jitter)
   Download:    17.10 Mbps (data used: 22.0 MB)                               
     Upload:    85.62 Mbps (data used: 89.6 MB)                               
Packet Loss:     0.0%

So my 'router' is still greater than a multiple of 10 less rapid than when outside of the VPN, and the desktop is greater than a multiple of 3.5 faster than the 'router'. According to speetest.net, the Comae server is about 750mi (1,200km) away from my ISP's server. Are these the kind of figures you'd expect?

[Staff 10334]

@dr_kristau

Hello!

it's strange that the router has a peak of 85 Mbit/s in upload and 17 in download. It makes us think about traffic shaping in download. Do you have your ISP traffic shaping (or traffic management) policy?

By the way, with that configuration OpenVPN 2.5 will not negotiate CHACHA20 on the Data Channel. If you check the log, you should see that you're still with AES-256-GCM. Modify in the following way:

• delete line cipher CHACHA20-POLY1305
• add lines:

data-ciphers CHACHA20-POLY1305
data-ciphers-fallback AES-256-CBC

Check in the log that OpenVPN 2.5 uses CHACHA20 in the Data Channel. You should see:

Quote

... Outgoing Data Channel: Cipher 'CHACHA20-POLY1305' initialized with 256 bit key
... Incoming Data Channel: Cipher 'CHACHA20-POLY1305' initialized with 256 bit key

Quote

According to speetest.net, the Comae server is about 750mi (1,200km) away from my ISP's server. Are these the kind of figures you'd expect?

Yes. 310 Mbit/s with an i7 and AES-256-GCM is expected. It means 620 Mbit/s on the server and it's more or less what we detect from a fiber line in Italy. We managed to beat that performance on the client side toward our VPN servers only from dedicated lines of dedicated servers.

Even 85 Mbit/s with the Celeron in AES-256-GCM sound quite reasonable. It remains to be seen why the router in download becomes so sluggish (17 Mbit/s instead of 85 Mbit/s). Finally, feel free to let us know the performance you will get with CHACHA20, we're curious to see what happens with a Celeron.

Kind regards
 

[dr_kristau 6]

22 minutes ago, Staff said:

it's strange that the router has a peak of 85 Mbit/s in upload and 17 in download. It makes us think about traffic shaping in download. Do you have your ISP traffic shaping (or traffic management) policy?

No, I don't have my ISP's traffic management policy to hand at the moment, I'm afraid. I believe that the upload/download inversion of my 'routers' numbers is an artifact produced my setup here, whereby the upload figure is in fact the download one.
 

22 minutes ago, Staff said:

By the way, with that configuration OpenVPN 2.5 will not negotiate CHACHA20 on the Data Channel. If you check the log, you should see that you're still with AES-256-GCM. Modify in the following way:

Okay, thank you. I will recheck and run the test again.

You might also be interested in the Wireguard comparison - I eventually got a successful installation of algo on a DigitalOcean droplet to work, on a server which must be about the same distance away from me as Comae. On the i7 I got:

        ISP: DigitalOcean, LLC
    Latency:    53.85 ms   (0.36 ms jitter)
   Download:   531.08 Mbps (data used: 610.4 MB)                               
     Upload:    24.53 Mbps (data used: 41.1 MB)                               
Packet Loss:     0.5%

And on the Celeron:

        ISP: DigitalOcean, LLC
    Latency:    53.31 ms   (0.33 ms jitter)
   Download:   347.76 Mbps (data used: 562.7 MB)                               
     Upload:    13.48 Mbps (data used: 20.3 MB)                               
Packet Loss:     1.0%

I'll run the CHACHA20 tests soon and post the results here.

[Staff 10334]

@dr_kristau

Thanks, let us know.

With Wireguard the performance loss you get with CHACHA20 in an AES-NI supporting CPU is more than compensated by the fact that Wireguard runs in the kernel space, while OpenVPN in the userspace (and does not scale in multicore processors). You will not get the same Wireguard performance with OpenVPN CHACHA20 in your Celeron, because of that very fact. In our infrastructure, anyway, even with Wireguard, you should not expect more than 400-500 Mbit/s, because no server usually has more than 800-900 Mbit/s free (things might change in the future with 10 Gbit/s line per single server).

Kind regards
 

[dr_kristau 6]

I may have misunderstood you with respect to my data plan - I have an asymmetric FTTP line contracted, 1000Mbps download and 300Mbps upload.

I followed you instructions, and double-checked the logs:

2020-10-30 22:03:52 Data Channel: using negotiated cipher 'CHACHA20-POLY1305'
2020-10-30 22:03:52 Outgoing Data Channel: Cipher 'CHACHA20-POLY1305' initialized with 256 bit key
2020-10-30 22:03:52 Incoming Data Channel: Cipher 'CHACHA20-POLY1305' initialized with 256 bit key

Then on the i7 I got:

    Latency:    67.61 ms   (147.14 ms jitter)
   Download:   400.52 Mbps (data used: 573.2 MB)                               
     Upload:     3.67 Mbps (data used: 5.6 MB)                               
Packet Loss:     7.1%

And on the Celeron:

    Latency:    67.28 ms   (0.32 ms jitter)
   Download:    77.75 Mbps (data used: 111.0 MB)                               
     Upload:     2.98 Mbps (data used: 2.6 MB)                               
Packet Loss:     8.7%

Please bear in mind that the network here is slower now than it was earlier - the neighbors must be watching Netflix or whatever (I'm UTC+1). This being so, I think we can see a greater difference with the i7 than with the Celeron.
 

15 minutes ago, Staff said:

With Wireguard the performance loss you get with CHACHA20 in an AES-NI supporting CPU is more than compensated by the fact that Wireguard runs in the kernel space, while OpenVPN in the userspace (and does not scale in multicore processors). You will not get the same Wireguard performance with OpenVPN CHACHA20 in your Celeron, because of that very fact.

That is the killer difference, as far as I also understand - Wireguard is now an in-tree kernel module as of 5.6. As far as I can see, running algo on a DigitalOcean droplet has the advantage of speed, but has the disadvantage primarily of price (we total a 700GiB transfer a month). The privacy aspect isn't such a concern for us, as although we torrent I can't imagine DigitalOcean receiving requests from law enforcement for such activity. I might be wrong, however.

[Staff 10334]

@dr_kristau

Hello!

Thanks for the detailed report.

Yes, Wireguard with CHACHA20 peaks 541 Mbit/s whereas OpenVPN peaks 400 Mbit/s.

We can't explain why, in OpenVPN, CHACHA20 is faster than AES even in an i7 (which supports AES-NI). AES should be faster. Maybe it was due only to the momentary network heavy usage you mention? Is the library linked by OpenVPN enabled to AES-NI (if it's OpenSSL, it should be by default in any recent version)?

Kind regards
 

[dr_kristau 6]

2 hours ago, Staff said:

We can't explain why, in OpenVPN, CHACHA20 is faster than AES even in an i7 (which supports AES-NI). AES should be faster. Maybe it was due only to the momentary network heavy usage you mention?

Doing these tests over a public network is far from ideal - I should really construct an OpenVPN server and a Wireguard/algo server locally for more precise measurements. Anyhow, I've just run the public tests again on the i7:
Outside VPN:

    Latency:     2.71 ms   (0.18 ms jitter)
   Download:   917.72 Mbps (data used: 449.7 MB)                               
     Upload:   290.37 Mbps (data used: 242.3 MB)                               
Packet Loss:     0.0%

Using CHACHA20 (Comae):

    Latency:    67.48 ms   (0.22 ms jitter)
   Download:   369.76 Mbps (data used: 547.5 MB)                               
     Upload:    71.57 Mbps (data used: 114.0 MB)                               
Packet Loss:     1.7%

Using AES (Comae):

    Latency:    67.55 ms   (0.12 ms jitter)
   Download:   571.38 Mbps (data used: 908.6 MB)                               
     Upload:    85.41 Mbps (data used: 152.5 MB)                               
Packet Loss:     1.7%

So yes, it must have been the heavy usage later in the evening which skewed yesterday's results.

Could I please request that Mekbuda be included in the CHACHA20 experimental configuration? This would give the Celeron more of a chance to obtain higher speeds, given that this server is less than 500km from me, and given that CHACHA20 gives the Celeron its best shot at higher encryption rates.

Thank you for your time!

[Staff 10334]

@dr_kristau

Good! And congratulations, 571 Mbit/s is a new, absolute, all time client record in our service! You and our server Comae managed to beat Wireguard performance too, that's really something.

OpenVPN 2.5 has been released a couple of days ago. Deployment on all servers has been completed but before restarting them all we need to resolve a very annoying problem: apparently a bug which was not there in any beta version and in RC1, but appeared now in the final release. The bug compromises CHACHA20 negotiation on the Data Channel between OpenVPN3 and OpenVPN 2.5 (but it's fine between OpenVPN 2.5<->OpenVPN 2.5).

Once we find out, isolate and fix the bug (either on client or server side) we will activate OpenVPN 2.5 on all servers and you will be able to use CHACHA20 on all of them. It's a matter of days, we hope.

Kind regards



 

[dr_kristau 6]

1 hour ago, Staff said:

Good! And congratulations, 571 Mbit/s is a new, absolute, all time client record in our service! You and our server Comae managed to beat Wireguard performance too, that's really something.

Glad to have been the one who made the test!
 

1 hour ago, Staff said:

Once we find out, isolate and fix the bug (either on client or server side) we will activate OpenVPN 2.5 on all servers and you will be able to use CHACHA20 on all of them. It's a matter of days, we hope.

I look forward to it.

It is interesting this 'instruction sets integrated into processors' versus 'modules integrated into kernels' - I hadn't considered the issue until today. Thanks again!
 

[Shiver Me Whiskers 15]

571 new record ?

I think I can do a bit better 😉

Server is 213.152.162.89, aka Miram

I think it's safe to say that this is the fastest commercial VPN in the world right now 😉

The numbers are so big that all measurements are hitting counting errors... but the reality is that it actually crossed 95 MBytes /second in the software I used ... and this measurement doesn't lie:

@Staff Time to bring those 10gbit servers 😉
1.0gbit is not enough anymore... hehe !

Here's another one, even faster:
https://www.speedtest.net/result/10444688806


This is not a random fluke, it really works this fast !