Can the 10G Full-Duplex Servers Operate at Nearly or Full Bandwidth/Capacity?

[ScanFarer 12]

I'm curious about whether users can truly max out the operational capacity of the 10G Full-Duplex Servers. In my experience, the Haedus server in New York rarely surpasses 3.5G out of its 10G bandwidth rating, and this observation holds true for other 10G servers as well. Even at its peak, when I employ multiple connections for seeding or torrenting, the speeds don't seem to reach their full potential or remain consistent.

I'm referring to the use of multiple connections in a manner similar to employing several simultaneous TCP streams to increase download/upload speeds from a server. Interestingly, there was a point when I could seed/upload at a continuous gigabit speed on the Haedus server. However, this experience isn't limited to torrenting; I've conducted tests using Iperf and other high-bandwidth usage scenarios. Strikingly, I find that I can use the most bandwidth when torrenting, presumably due to the ability to have several thousand leeches connected. I consistently get about 500 - 700 Mbps on the Haedus server using Wireguard.

I can't help but wonder why I haven't seen more bandwidth being utilized, especially with the increasing number of users on the server. 

[Staff 10015]

@ScanFarer

Hello!

Your analysis is correct. The maximum amount of bandwidth this month which Haedus could give was 7 Gbit/s:


One of the main reasons is that the load on the CPU increases more than linearly as the number of OpenVPN connected clients increases. Other reasons include routing, since the maximum throughput one can get is the maximum throughput that the slowest hop in the route can provide.

While more and more users switch to WireGuard, the servers will be able to deliver more bandwidth, because WireGuard scales excellently. Consider that each OpenVPN process runs in a single thread of a single core (when OpenVPN's DCO is stable things will change), forcing us to run multiple processes, while we only need to run one WireGuard process which distributes load evenly on all cores. Furthermore, WireGuard doesn't copy data from kernel to userspace and vice-versa, while OpenVPN does (again, OpenVPN's DCO running in the kernel space will do the same). Actually, Haedus can provide almost 4 Gbit/s (8 Gbit/s on the server, in practice) when the connection is performed by a single client. Increasing MTU size might also improve performance further, only provided that there's room in the frames.

From Android devices, almost everyone runs WireGuard, because Eddie Android edition is set to WireGuard by default. From desktop devices, Eddie's default is OpenVPN at the moment, so many users start with that default setting and stick to it. What to offer by default is a matter of analysis on desktop systems: WireGuard poses privacy issues and can be easily blocked, while OpenVPN is definitely and sometimes significantly slower even on most AES-NI supporting devices but offers dynamic address assignment and the invaluable abilities to connect over stunnel, SSH, SOCKS proxies, Tor (working in TCP).

Kind regards
 

[ScanFarer 12]

@Staff

Since we are seeing that this server is nearly at its limit as far as connected users and bandwidth usage, can we expect to see another 10 Gigabit server soon?
I got a message from Staff in August suggesting that news about (10G) server expansion would be contingent upon userbase growth. Initially, there were indications of updates by November. Just wondering if there were any recent developments on this matter. I want to avoid appearing overly pushy, but I am genuinely interested in gaining some insight on the progress.

[Staff 10015]

4 hours ago, ScanFarer said:

@Staff

Since we are seeing that this server is nearly at its limit as far as connected users and bandwidth usage, can we expect to see another 10 Gigabit server soon?
I got a message from Staff in August suggesting that news about (10G) server expansion would be contingent upon userbase growth. Initially, there were indications of updates by November. Just wondering if there were any recent developments on this matter. I want to avoid appearing overly pushy, but I am genuinely interested in gaining some insight on the progress.

Hello!

Yes, definitely. Stay tuned and follow the "News" AirVPN forum for updates. 😉

Kind regards

[Staff 10015]

@ScanFarer

Hello!

New 10 Gbit/s full duplex server has been added in Los Angeles. Another server with high bandwidth has been added in Denver (Colorado) with a 10 Gbit/s interface and port and 3 Gbit/s bandwidth (full duplex) guaranteed, burstable to 10 Gbit/s.
https://airvpn.org/status

Kind regards
 

[foDkc4UySz 9]

On 12/8/2023 at 9:12 AM, Staff said:

One of the main reasons is that the load on the CPU increases more than linearly as the number of OpenVPN connected clients increases.

While more and more users switch to WireGuard, the servers will be able to deliver more bandwidth, because WireGuard scales excellently.
 

So why not just make WireGuard Eddie's default protocol?

I switched a year ago from OpenVPN tunneling simply being too slow to my very-nearest preferred country-pool at peak times, to WireGuard with the service experience improvements based solely from a connection-speed perspective being like turning the lights on.

There maybe a (significant?) proportion of users whom dare not fiddle with Eddie's configuration interface, due to it's technical nature. Let the geek's do that, if the geek's want OpenVPN.

In the mean time, why let your service and reputation suffer, when all the next Eddie upgrade need change is "if network protocol  == automatic, select WireGuard".

Please, yes, continue to add more 10gBit/s servers wherever locations are privacy-philosophically and price-contractually feasable, but you've just admitted bandwidth achievement rates are limited because of OpenVPN... Eddie's default protocol.

You've the ability the maximise server bandwidth capacity, improve customer connection-speed experience, and the change is entirely within your control for those who dare not enter into the configuration interface to choose OpenVPN, or build their own OpenVPN Config.
 

[Staff 10015]

@foDkc4UySz

Hello!

WireGuard is the default choice in Eddie Android edition and it will be in the next AirVPN Suite for Linux (you can see the AirVPN Suite 2.0.0 preview version which is already supporting WireGuard fine). On Eddie Desktop edition the matter will be thoroughly discussed. On one hand you have the poor performance on networks shaping WireGuard, the complete WireGuard block in countries which are not irrelevant for AirVPN, together with the privacy problems posed by WireGuard. On the other hand you have the superior performance of WireGuard in agnostic networks and the fact that the privacy problems are mitigated by our setup and can be resolved by your behavior (key renewals). Plus, it must be taken into consideration the fact that if UDP is de-prioritized, then even the default OpenVPN on UDP of Eddie Desktop edition will suffer just like WireGuard.

Kind regards
 

[ms2738 5]

On 12/8/2023 at 1:12 AM, Staff said:

While more and more users switch to WireGuard, the servers will be able to deliver more bandwidth, because WireGuard scales excellently. 

If Wireguard scales and allows full use of server bandwidth, had you considered Wireguard only servers?  Seems like that would make sense at least on the 10G so as not to waste so much capacity?

[ScanFarer 12]

18 hours ago, ms2738 said:

If Wireguard scales and allows full use of server bandwidth, had you considered Wireguard only servers?  Seems like that would make sense at least on the 10G so as not to waste so much capacity?

I know Mullvad already does this, they separate there OpenVPN and Wireguard protocols to different servers.

[Staff 10015]

@ms2738
@ScanFarer

Hello!

DCO would resolve any problem and would not require this relevant waste of resources with the infrastructure split into "WireGuard only" and "all the rest" servers. As the "kernel acceleration module" which then became "DCO" project was announced in 2019 we thought the split you suggest would have been unnecessary. Five years have passed since that announcement and DCO is still "under heavy development, therefore neither its userspace API nor the code itself is considered stable and may change radically over time." so we might review our plans. Let's see anyway whether this first quarter of 2024 brings news on DCO development.

Kind regards
 

[ms2738 5]

On 1/17/2024 at 5:19 AM, Staff said:

@ms2738
@ScanFarer

Five years have passed since that announcement and DCO is still "under heavy development, therefore neither its userspace API nor the code itself is considered stable and may change radically over time." so we might review our plans.

it seems like Wireguard is more and more the solution of choice.  Even with updates like DCO it seems bloated and outdated compared to Wireguard.  I think it's great you support both protocols, and wish you supported more, but I wonder if you really need to support both on every server?

Maybe set up some Wireguard only servers in places you have multiple servers just as a test... similar as you're doing with OpenVPN DCO?

[Staff 10015]

6 hours ago, ms2738 said:

it seems like Wireguard is more and more the solution of choice.  Even with updates like DCO it seems bloated and outdated compared to Wireguard.  I think it's great you support both protocols, and wish you supported more, but I wonder if you really need to support both on every server?

Hello!

Yes, it's strictly mandatory to support OpenVPN on every and each existing server, otherwise we would break compatibility with a relevant portion of customers who rely on OpenVPN support for their devices (or even because WireGuard is blocked in their network) on specific server(s). Stopping OpenVPN on existing server(s) would be a huge and serious problem, even without considering the contractual breaches.

Kind regards
 

[ms2738 5]

1 hour ago, Staff said:

Hello!

Yes, it's strictly mandatory to support OpenVPN on every and each existing server, otherwise we would break compatibility with a relevant portion of customers who rely on OpenVPN support for their devices (or even because WireGuard is blocked in their network) on specific server(s). Stopping OpenVPN on existing server(s) would be a huge and serious problem, even without considering the contractual breaches.

Kind regards
 

You could certainly ADD a Wireguard only server, without taking away an OpenVPN server.

[Staff 10015]

6 minutes ago, ms2738 said:

1 hour ago, Staff said:

Stopping OpenVPN on existing server(s)

You could certainly ADD a Wireguard only server, without taking away an OpenVPN server.

Hello!

The matter for future servers might be different of course.

Kind regards
 

[ms2738 5]

1 hour ago, Staff said:

The matter for future servers might be different of course.

That's what I was thinking. and there's no question that all of the market share growth has been Wireguard for several years.  I think it's fair to say Wireguard won.  Do you see people going back to OpenVPN when they get DCO working and if so why?

If you added just a few Wireguard only servers (or even just one) around the world to test how much more performant they'd be would probably be very telling?  Might even get you some press... even better than advertising?

Thanks you for your consideration

[Staff 10015]

3 hours ago, ms2738 said:

That's what I was thinking. and there's no question that all of the market share growth has been Wireguard for several years.  I think it's fair to say Wireguard won.  Do you see people going back to OpenVPN when they get DCO working and if so why?

Hello!

If DCO becomes stable and maintains the performance claimed in the OpenVPN Hackathon 2019 yes, we do, as DCO would outperform WireGuard (in real life we don't see much difference though, but we must keep into account that DCO is still highly experimental). On top of multi-threading support, it could be used over TCP too, which would be significant. On the server side it resolves also WireGuard limitation regarding the automatic IP address assignment in place of the moronic static, hard coded and on file key<->address bijection needed by WireGuard. On the other hand, on the Linux universe, we don't foresee DCO included in the mainline kernel, while WireGuard is already there. We'll see.
 

Quote

if you added just a few Wireguard only servers (or even just one) around the world to test how much more performant they'd be would probably be very telling?

It's a worthwhile test and we're thinking about it.

Kind regards
 

[benfitita 39]

On 1/19/2024 at 11:44 AM, ms2738 said:

Do you see people going back to OpenVPN when they get DCO working and if so why?

I have returned to OpenVPN on my router. For an unknown reason my ISP, or some exchange along the route, has some kind of an intermittent WireGuard problem during the evening internet rush hours. From time to time wg connections are getting frozen and can't be reestablished.

I appreciate that I can easily switch between protocols on any server. Unless there's a specific benefit from wg-only servers, I'd prefer to keep the current great flexibility that AirVPN provides.

[ms2738 5]

9 hours ago, benfitita said:

Unless there's a specific benefit from wg-only servers, I'd prefer to keep the current great flexibility that AirVPN provides.

The primary reason is that OpenVPN is single-threaded and CPU bound preventing utilization of available bandwidth:

"the load on the CPU increases more than linearly as the number of OpenVPN connected clients increases... each OpenVPN process runs in a single thread of a single core, forcing us to run multiple processes, while we only need to run one WireGuard process which distributes load evenly on all cores. Furthermore, WireGuard doesn't copy data from kernel to userspace and vice-versa, while OpenVPN does".

Unfortunately OpenVPN doesn't just limit those who choose to use OpenVPN, it's limiting the bandwidth potential for the whole server.  "While more and more users switch to WireGuard, the servers will be able to deliver more bandwidth, because WireGuard scales excellently."  In the mean time, should AirVPN keeping buying more bandwidth no one can use?

FWIW, "intermitent Wireguard problem" doesn't sound right.  What port are you using?

[ms2738 5]

@ScanFarer said "I'm curious about whether users can truly max out the operational capacity of the 10G Full-Duplex Servers. In my experience, the Haedus server in New York rarely surpasses 3.5G out of its 10G bandwidth rating, and this observation holds true for other 10G servers as well. Even at its peak, when I employ multiple connections for seeding or torrenting, the speeds don't seem to reach their full potential or remain consistent.

I'm referring to the use of multiple connections in a manner similar to employing several simultaneous TCP streams to increase download/upload speeds from a server. Interestingly, there was a point when I could seed/upload at a continuous gigabit speed on the Haedus server. However, this experience isn't limited to torrenting; I've conducted tests using Iperf and other high-bandwidth usage scenarios. Strikingly, I find that I can use the most bandwidth when torrenting, presumably due to the ability to have several thousand leeches connected. I consistently get about 500 - 700 Mbps on the Haedus server using Wireguard.

I can't help but wonder why I haven't seen more bandwidth being utilized, especially with the increasing number of users on the server."

I recently asked almost the same thing, it's clear the higher bandwidth servers are being under utilized, and it seems to be due to OpenVPN.

@Staff aren't yet open to fixing this with Wireguard only servers, but I wonder if limiting OpenVPN connections based on CPU usage would solve the problem?  Being protocol neutral is great, except when it limits the efficiency of the network for everyone.  It's hard not to feel like OpenVPN, with their five year old promises, is ruining it for the rest of us.

[benfitita 39]

11 hours ago, ms2738 said:

The primary reason is that OpenVPN is single-threaded and CPU bound preventing utilization of available bandwidth:

"the load on the CPU increases more than linearly as the number of OpenVPN connected clients increases... each OpenVPN process runs in a single thread of a single core, forcing us to run multiple processes, while we only need to run one WireGuard process which distributes load evenly on all cores. Furthermore, WireGuard doesn't copy data from kernel to userspace and vice-versa, while OpenVPN does".

Unfortunately OpenVPN doesn't just limit those who choose to use OpenVPN, it's limiting the bandwidth potential for the whole server.  "While more and more users switch to WireGuard, the servers will be able to deliver more bandwidth, because WireGuard scales excellently."  In the mean time, should AirVPN keeping buying more bandwidth no one can use?

FWIW, "intermitent Wireguard problem" doesn't sound right.  What port are you using?

OpenVPN is only limiting WireGuard if servers don’t have fast enough CPUs to handle extra OpenVPN overhead. There could be other hardware and software (kernel) limitations coming from all that userspace - kernel switching. I’m sure @Staff has all the information to make the right calls about optimizing this. 

WireGuard has this problem for me on all ports. This affects other VPN providers as well. Problem isn’t on my end because it happens on various clients and routers I tried. On the other hand, OpenVPN works for me on any port. Go figure. I wouldn’t want to hijack this topic, so let’s wrap it up. 

[legitacc1234 5]

On 1/26/2024 at 10:02 PM, ms2738 said:

Being protocol neutral is great, except when it limits the efficiency of the network for everyone.  It's hard not to feel like OpenVPN, with their five year old promises, is ruining it for the rest of us.

I guess 99% of the users are fine with the speed, including me. Are you having reallife scenario problems with the speed, or is it just numbers that arnt right for you? I use AirVPN 24/7 and never notice it. Only thing i notice is captchas, but thats with every vpn and nothing they can do about it. i normally get 500-600 on the netherlands servers from my 1 gb connection and i know many who get similar results from European servers.As long as i dont notice a single problem with speeds with AirVPN, i dont even bother looking at numbers or anything else, since i have zero problems realworld and numbers are often highly misleading. 

[oassQ9w4cbl4AySZhhth%p36x 3]

it would be really nice to get a wireguard only NL and US server - i'm only on gigabit but it feels like dalim at times struggles because of ovpn users that exist on it. 

it would be nice for the total users values on the dashboard to be split into OVPN / WG to give more insightful heuristics into how much OVPN opters are slowing things down

[Staff 10015]

On 1/26/2024 at 10:02 PM, ms2738 said:

limiting OpenVPN connections based on CPU usage would solve the problem?

Hello!

Server load is indeed, and it has always been, one of the variables which are taken into calculation to determine the server rating and therefore recommend a server over another. Furthermore we have other inner variables which are monitored and are indicative of a server overload. In extreme cases yes, a server may be even closed to new connections due to overload, nothing new here. Such events are quite rare.
 

4 hours ago, oassQ9w4cbl4AySZhhth%p36x said:

more insightful heuristics into how much OVPN opters are slowing thingsdown

Where does this assumption come from? There's no evidence that OpenVPN users are slowing down WireGuard users. The problem is mainly on the OpenVPN client side, as users running OpenVPN on agnostic networks and/or on devices not supporting AES-NI might be unable to reach the performance which they would enjoy with WireGuard.

Again, whether or not prioritizing WireGuard over OpenVPN on the client side through our software default settings is not a problem due to OpenVPN stealing CPU time to WireGuard on server side (as incorrectly understood by @ms2738 too, we see now), as all servers have plenty of CPU available time for WireGuard, but a problem of customer satisfaction (in our previous message we mentioned the pros and cons which make the choice non trivial).

Kind regards
 

[oassQ9w4cbl4AySZhhth%p36x 3]

The assumption is just speculation for the most part, if you are saying it is not the case then I have no other rebuttal to offer and will take it at face value. It does make some sense though that the inefficiency with ovpn when done en mass would result in slowdowns and total throughput limitations that are not line speed due to the additional time it takes in userland. 

measuring CPU in general is always a tough thing, its never as simple as just looking at the load counter which you well know. interrupts play a huge part in it, HT / SMT, CCDx / NUMA traversal and more. We know in general that ovpn is quite bloated and inefficient when compared to wireguard so the assumption at first glance doesn't seem without merit.

[ms2738 5]

6 hours ago, Staff said:

"is not a problem due to OpenVPN stealing CPU time to WireGuard on server side (as incorrectly understood by @ms2738 too, we see now)"
 

Hey - that was not my understanding at all.  However, is it not true that the total bandwidth utilization on 10gb servers is currently CPU limited due to OpenVPN?  In other words, if the 10Gb servers were Wireguard only they'd have more bandwidth to end users?