openvpn-dco - data channel offload

[qitorin 1]

Does Airvpn have any intention to implement openvpn--dco to accelerate transmission speeds. It is possible to install and test kernel module for such a purpose but Eddie at least complains - "OpenVPN > Note: --data-cipher-fallback with cipher 'AES-256-CBC' disables data channel offload". Openvpn-dco would at least be of great benefit for router based openvpn connection and would keep openvpn competitive with wireguard in the speed front. On top of that it should be considered that openvpn connections are more robust and have more options to choose from. Thanks for any info.
 

[OpenSourcerer 1359]

AFAIK DCO is an OpenVPN 2.6 feature which is still in development. Running betas in a productive setting is ill-advised.
Though, I guess, you asked in case it gets released, right? This I cannot answer you.

[oassQ9w4cbl4AySZhhth%p36x 2]

oh nice someone also made a thread about this. 

I hope AirVPN plan to implement it in the future. It seems like a great way to avoid all the context switching and slower speeds of openvpn vs wireguard. 

[oassQ9w4cbl4AySZhhth%p36x 2]

https://openvpn.net/blog/openvpn-data-channel-offload/
further to this, looks like any openvpn 2.6.0+ will natively support it?

wonder if staff have any insight into timeline for upgrades

[Flx 76]

From Staff--->>"Now that OpenVPN 2.6.0 has been released, we will progressively implement it in our servers, starting from mid-February."

[fysh 3]

On 2/6/2023 at 8:01 PM, Flx said:

From Staff--->>"Now that OpenVPN 2.6.0 has been released, we will progressively implement it in our servers, starting from mid-February."

@Staff any news on when 2.6.0 will start to roll out?

[Staff 9751]

1 hour ago, fysh said:

@Staff any news on when 2.6.0 will start to roll out?

Hello!

We have experienced a serious kernel panic problem apparently reported here:
https://github.com/OpenVPN/ovpn-dco/issues

The reported kernel crashes which we also experienced are fearsome, because when they occur all users (even WireGuard ones of course) are disconnected, machine control is lost, and a reboot via IPMI or hard power cycle is required. Before allowing something to run as a kernel part on a production server for all of you, that something must behave more properly. We will keep testing and searching for a stable setup, and of course we also hope in imminent bug fixes. We will keep you informed. Of course rolling out OpenVPN 2.6 without DCO is an option, but we would prefer to migrate directly to OpenVPN with DCO, instead of having to do it in two different steps.

Kind regards
 

[fysh 3]

Completely understandable, thanks for the explaination!

Hope it can be resolved soon.

[oassQ9w4cbl4AySZhhth%p36x 2]

On 3/12/2023 at 5:14 PM, Staff said:

Hello!

We have experienced a serious kernel panic problem apparently reported here:
https://github.com/OpenVPN/ovpn-dco/issues

The reported kernel crashes which we also experienced are fearsome, because when they occur all users (even WireGuard ones of course) are disconnected, machine control is lost, and a reboot via IPMI or hard power cycle is required. Before allowing something to run as a kernel part on a production server for all of you, that something must behave more properly. We will keep testing and searching for a stable setup, and of course we also hope in imminent bug fixes. We will keep you informed. Of course rolling out OpenVPN 2.6 without DCO is an option, but we would prefer to migrate directly to OpenVPN with DCO, instead of having to do it in two different steps.

Kind regards
 

hoping this is implemented sometime soon, would be really nice to see what speeds are capable with DCO enabled. If theres any beta testing available I can provide some. 

[Staff 9751]

@oassQ9w4cbl4AySZhhth%p36x

We will keep you posted. Unfortunately, it's possible that in an initial stage DCO will be only on experimental servers, so yes, persons available to testing will be invaluable, thanks! Check here for a quick update:
https://airvpn.org/forums/topic/56119-new-10-gbits-server-available-bg/

Kind regards
 

[oassQ9w4cbl4AySZhhth%p36x 2]

On 3/24/2023 at 11:24 AM, Staff said:

@oassQ9w4cbl4AySZhhth%p36x

We will keep you posted. Unfortunately, it's possible that in an initial stage DCO will be only on experimental servers, so yes, persons available to testing will be invaluable, thanks! Check here for a quick update:
https://airvpn.org/forums/topic/56119-new-10-gbits-server-available-bg/

Kind regards
 

hi any update on this? seems some new builds which resolved the issues mentioned have been made available according to the developer in the link you provided

[OpenSourcerer 1359]

I was able to test this a bit. Version 0.2.20230323-5-g961c60d on kernel 6.2.10 seems to work. But I needed to ignore any pushed compression options. As all servers push --comp-lzo no to clients, it must be ignored with --pull-filter:

pull-filter ignore comp-lzo

Oh, and I needed to replace --comp-lzo in the config file with an explicit --allow-compression no:

allow-compression no

.

$ sudo openvpn file.ovpn  
2023-04-15 15:32:19 OpenVPN 2.6.2 [git:makepkg/3577442530eb7830+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO] built on Mar 24 2023
2023-04-15 15:32:19 library versions: OpenSSL 3.0.8 7 Feb 2023, LZO 2.10
2023-04-15 15:32:19 DCO version: 0.2.20230323-5-g961c60d
[…]
2023-04-15 15:32:20 PUSH: Received control message: 'PUSH_REPLY,comp-lzo no,redirect-gateway  def1 bypass-dhcp,dhcp-option DNS 10.26.234.1,route-gateway 10.26.234.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.26.234.233 255.255.
255.0,peer-id 0,cipher AES-256-GCM'
2023-04-15 15:32:20 Pushed option removed by filter: 'comp-lzo no'
[…]
2023-04-15 15:32:20 Initialization Sequence Completed

.

[oassQ9w4cbl4AySZhhth%p36x 2]

Nice job OpenSourcerer, what is the performance like?

Its a shame that AirVPN have decided to ignore providing a status update, they are usually very forthcoming with responses. I guess all companies change though.

[dIecbasC 38]

1 hour ago, oassQ9w4cbl4AySZhhth%p36x said:

It’s a shame that AirVPN have decided to ignore providing a status update, they are usually very forthcoming with responses. I guess all companies change though.

there’s one from less than four weeks ago just above? 

[OpenSourcerer 1359]

10 hours ago, oassQ9w4cbl4AySZhhth%p36x said:

Nice job OpenSourcerer, what is the performance like?

Absolutely nonexisting, let me tell you. It must mature way, way more. Will be quite some time until we see it in the kernel.
 

10 hours ago, oassQ9w4cbl4AySZhhth%p36x said:

Its a shame that AirVPN have decided to ignore providing a status update, they are usually very forthcoming with responses. I guess all companies change though.

What's that supposed to mean? It was written over and over again that they'll keep everyone posted. Be careful with your assessment…

[Staff 9751]

On 4/18/2023 at 7:24 AM, oassQ9w4cbl4AySZhhth%p36x said:

Nice job OpenSourcerer, what is the performance like?

Its a shame that AirVPN have decided to ignore providing a status update, they are usually very forthcoming with responses. I guess all companies change though.

Hello!

Please follow the thread here: https://airvpn.org/forums/topic/56119-new-10-gbits-server-available-bg/

The message by Antonio Quartulli on the end of March implies that our tests had to re-start almost from scratch with the new DCO and the new OpenVPN 2.6.2. Remember that DCO, as reported in the GitHub repository https://github.com/OpenVPN/ovpn-dco

ovpn-dco is currently under heavy development, therefore neither its userspace API
nor the code itself is considered stable and may change radically over time.

Remember also that every time the code changes radically our tests must re-start almost from scratch.

About the stability, with 2.6.2 and latest current experimental DCO 0.2 module we have not had anymore kernel panics. Given the above you can easily understand that DCO is not ready for production on the server side of course, so we're planning to put a highly experimental server to let the community test with no stability or backward compatibility commitments.

For latest issues and updates about them, please follow GitHub as well:
https://github.com/OpenVPN/ovpn-dco/issues

Kind regards
 

[NaDre 154]

Some time ago AirVPN implemented a load balancer for their OpenVPN servers:

https://airvpn.org/forums/topic/30039-airvpn-ip-keeps-changing/?tab=comments#comment-78778

This was because the OpenVPN server is single-threaded/uses a single CPU. It had the effect that the local private IP address assigned could change even though you used the same address to connect. It depended on which core/server you got assigned to.

Does AirVPN still do this? If so will this continue once DCO is rolled out on their servers?

I found a comment about some effects DCO will have on load balancing here:

https://forums.openvpn.net/viewtopic.php?t=34907
 

Quote

...

Post by openvpn_inc » Sat Nov 05, 2022 3:58 pm

Hi diz,

With large numbers of clients, you run up against the issue of CPU overload. The openvpn(8) 2.5.x and earlier software is a single-threaded application, so your server process is running on a single CPU core.

Version 2.6 (now available only through git) addresses this limitation with a new feature called Data Channel Offload (DCO). With DCO the main server thread is still on that one CPU core, but the more challenging work of encrypting and decrypting and routing packets is shared among other CPU cores.

...

To directly answer your question, we don't recommend using load balancers. ...

My suggestion, given that you are already running a server for 1000+ clients, is to set up a server with 2.6 from git and DCO. ...
 

So it seems that the bulk of the load will be in kernel space and spread across cores. So will AirVPN still continue with their load balancing?
 

[Staff 9751]

Hello @NaDre

the DCO module might actually make the load balancing superfluous. OpenVPN 2.6.x alone will not, though. So we will keep the load balancing active in the meantime. It will make sense to disable it when DCO enters a stable phase and that stable release is tested on the field, showing that the load balancing is no more necessary. Our current load balancing allowed our servers to beat the OpenVPN limits, as you may remember. The current maximum throughput reached on 10+10 Gbit/s servers (all OpenVPN instances together, of course, and WireGuard excluded) is about 4 Gbit/s (2 Gbit/s + 2 Gbit/s). Without load balancing OpenVPN 2.5.5. and 2.6.2 (without DCO) on our most powerful CPUs, with AES-256-GCM and/or CHACHA20-POLY1305, single instance, can't beat 1.7 Gbit/s. (850 Mbit/s + 850 Mbit/s). OpenVPN + DCO promises to beat even that performance, on a level playing field.

They don't recommend load balancing but then they say:

Quote

The commercial OpenVPN Access Server software has a different workaround for this problem: it runs one daemon per protocol (TCP/UDP) per CPU core. And the network you assign to the VPN is split among each openvpn daemon.

<vanity mode on> which is something similar to what we do, except that our load balancing system is better than this as it uses directly the kernel to welcome clients and assign them to the proper OpenVPN instance <vanity mode off> 😉

Kind regards
 

[go558a83nk 352]

I just realized that Pfsense+ software (which is still free for home users) has DCO capability while the pfsense CE software does not.  So I'm migrating to it today in preparation for testing DCO.  I believe I'll have need for openvpn (again) in the future and would love an openvpn that is faster on my pfsense box with one of those new N100 CPUs.  Wireguard is blazing fast, BTW, and the power usage is tiny.

So, can't wait for a DCO test to begin here!

[dIecbasC 38]

Plus also supports QAT which with a suitable processor should improive OpenVPN performance and/or reduce CPU utilization. 

[go558a83nk 352]

29 minutes ago, dIecbasC said:

Plus also supports QAT which with a suitable processor should improive OpenVPN performance and/or reduce CPU utilization. 

good point but I don't think this little box supports it.  I'll find out.

[go558a83nk 352]

I was able to get DCO to connect with pfsense+ 23.05 (thanks to opensourcerer's notes about compression above) but no data actually transfers and after a bit I get a ping restart.  The client area session info also indicates no traffic moving.  Would something need to change on AirVPN's end?  My understanding that even if only the client is running DCO some improvement in throughput could be had so I figured it would just work without Air changing anything on their end.

[adrianlai2000@hotmail.com 0]

On 6/6/2023 at 8:09 AM, go558a83nk said:

I was able to get DCO to connect with pfsense+ 23.05 (thanks to opensourcerer's notes about compression above) but no data actually transfers and after a bit I get a ping restart.  The client area session info also indicates no traffic moving.  Would something need to change on AirVPN's end?  My understanding that even if only the client is running DCO some improvement in throughput could be had so I figured it would just work without Air changing anything on their end.

Same issue here