ANSWERED Very poor performance when set up on router

[LazyLizard14 11]

After some testing with the AirVPN client under Windows I proceed to set up the VPN connection on my router so all my computers can make use of it. But when the VPN tunnel is established from the router I only reach a speed of just about 7 MBit/s. When connecting with the Windows client (same port, protocol and server) i get about 30 MBit/s! The router quite powerful (Ubiquiti Edgerouter Lite) and not the bottleneck in terms of performance.

What could cause this behavior? MTU? On the router the vtun0 interface is up with MTU 1500.

I'm clueless and need some advise for troubleshooting

[Staff 9973]

Hello,

 

what is the router CPU? We can't easily find this information on the manufacturer's web site.

 

Kind regards

[LazyLizard14 11]

Hardware Info

Datasheet

Unlikely that the router is causing the slowdown, especially as the cpu load is floating well below 20% .

[zhang888 1066]

Hi,

If I found your model correctly,

 

http://dl.ubnt.com/datasheets/edgemax/EdgeRouter_Lite_DS.pdf

 

Processor Dual-Core 500 MHz, MIPS64 with Hardware Acceleration for Packet Processing

 

That means your router has a very slow CPU for a normal OpenVPN connection with AES-256.

Don't believe those marketing terms like "hardware acceleration", when nothing is specified it probably means nothing.

At least OpenVPN can support only OpenSSL ciphers, and OpenSSL supports only Padlock, AES-NI or Hexacore as accelerators for AES.

 

 

P.S.

Just to compare, the below < $99 range TP-Link, Asus, Linksys and Netgear routers come with about 700 MHz and can achieve 10-12Mbit, on DD-WRT or OpenWRT.

[mage1982 15]

OP, are you talking about megabits or megabytes? Because no program I've ever seen reports transfer speeds in megabits per second - on the other hand, seven megabytes per second seems not unreasonable for a router CPU like that, and lines up with what I've seen.

 

For comparison, I've been able to max out a 10 Mbit connection with OpenVPN running on an old desktop machine with a 450 MHz Pentium III, at about 20% CPU use if memory serves me right. Ten megabits connection, giving a one megabyte per second download speed.

 

(That is of course not directly comparable to an embedded MIPS CPU, but might give a ballpark estimate.)

 

So, just adding it up, I should have hit 100% CPU use at about 5-6 megabytes/second, and that is about where you are too. It even lines up with zhang888's results.

 

I think your router CPU is the bottleneck after all.

[LazyLizard14 11]

I really wonder why you all so easy to say the router is the problem? This router is capable of more than average consumer products in this price range.

cpu load of the router keeps well below 20% whole time and is NOWHERE even close to be maxed out. Same with RAM usage. I observed this, don't worry.

 

@mage1982: yes I am talking about 7 Megabit/s that I am getting only. With 7 Megabyte/s I would be more than happy for my 50 Megabit/s connection.

 

SG TCP/IP Analyzer gives me these results

 

TCP options string = 020405480103030801010402 MTU = 1392
MTU is not fully optimized for broadband. Consider increasing your MTU to 1500 for better throughput. If you are using a router, it could be limiting your MTU regardless of Registry settings. MSS = 1352
MSS is not optimized for broadband. Consider increasing your MTU value. Default TCP Receive Window (RWIN) = 66048
RWIN Scaling (RFC1323) = 8 bits (scale factor: 2^8=256)
Unscaled TCP Receive Window = 258

RWIN is not fully optimized. The unscaled RWIN value is lower than it should be. Also, RWIN being close to and above 65535 does not justify the header overhead of enabling TCP 1323 Options. You might want to use one of the recommended RWIN values below.

For optimum performance, consider changing RWIN to a multiple of MSS.
Other RWIN values that might work well with your current MTU/MSS:

[zhang888 1066]

I guess we just say it based on our humble experience. It's not something against your router, please don't take it personal

 

You said that you get 30Mbit on a Windows client, so this eliminates the ISP issue (A person in another thread also had a problem,

but he got it all the time no matter what equipment he used, for example).

 

So if we narrow it down, and look at the specs of your router, and a little Googling, we can see that you are not the only one who complains

about slow OpenVPN speeds on this model.

What you mean is that your router probably have more features, compared to others, but that doesn't necessarily means better speeds.

 

The hardware looks like an average 2011 home router, except the RAM which is not significant for a vpn client.

Since the OS is closed source, and the OpenVPN client in it as well, it can be problematic to tweak advanced settings on it.

 

Can you try setting the MTU to 1500?

[LazyLizard14 11]

Hello zhang888,

I not take it personally and of course agree with you thats way too often a cheap router struggles with the encryption and therefore causes poor performance. But I think this is not the case here as the CPU load is very low. My ISP isn't the problem either: without using vpn I get constantly 50 Mbit/s. Of course using vpn comes with some performance loss and probably even my ISP does some traffic shaping with encrypted traffic as well. But that isn't a plausible explanation either for only getting about 7Mbit/s. Correct me if I am wrong but setting up the tunnel on the router should give me the same speeds as using the Eddie client on a windows machine if a) the same protocol settings and server used and the router's cpu isn't maxed out. Right?

Thats how my interfaces look like:

All interfaces have a MTU of 1500 already. Don't mind the missing IP for vtun0: as its a virtual device IP isn't shown. But i verified by cli that it is connected properly.

Is it correct that they are all set to the same MTU size?

On a another thread support staff suggested tweaking the MTU with the mssfix directive in the OpenVPN config file instead of using tun-mtu, link-mtu or fragment. Would be good if support could help here in sorting out the MTU issues as it's likely that this causes the slowdown.

[userr1980 1]

250 kb/s sounds about right for a router that isn't capable of processing the encryption quickly enough.

[Staff 9973]

@LazyLizard14

 

Hello!

 

The problem is neither on your ISP nor on the VPN servers, considering that you wrote that a connection from a PC keeps 30 Mbit/s.  What MIPS64 CPU is in the router? It is not specified on the data sheet. Only some models of the MIPS64 CPU series have additional AES abilities. If your router CPU lacks that feature, probably the bottleneck is caused by the CPU, in spite of the low load you detect. Given the price range of your router, it is reasonable.

 

Kind regards

[LazyLizard14 11]

The router is equipped with a Cavium Octeon Plus CN5020 (500 MHz) processor.

See datasheet

 

Integrated coprocessors for application acceleration

•Packet I/O processing, QoS, TCP acceleration

•Support for IPsec, SSL, DH, SRTP, WLAN security,

 DES, 3DES, AES (up to 256-bit including GCM),

 SHA1, SHA-2 up to SHA-512, RSA, ECC, KASUMI,

 and Data-at-rest security (AES-XTS)

[zhang888 1066]

The crypto acceleration seems to be still on their "todo" list, for almost 2 years.

 

http://community.ubnt.com/t5/EdgeMAX/Features-incompatible-with-offload/m-p/395350#M4750

 

 

 

• The "cryptographic acceleration" is used to offload encryption/decryption operations etc. For example, as mentioned IPsec traffic is accelerated by offloading the encryption/decryption/etc. to the hardware cryptographic acceleration function. Not all cryptographic operations are currently accelerated since each application need to be tweaked use the hardware cryptographic acceleration. So for example, as mentioned OpenVPN is not yet using the acceleration function.
• Regarding the cryptographic acceleration and OpenVPN, as mentioned one of our TODO item is to "tweak" OpenVPN such that it can use hardware acceleration to offload the encryption/decryption operations. As with all other items on our TODO list, we certainly know the importance but it really depends on the availability of development resources and we don't have a time estimate at this point.

 

You can try flashing pfSense on it,

https://lists.pfsense.org/pipermail/list/2014-February/005455.html

 

At least with an open source system there might be a chance that FreeBSD got drivers for that chipset.

Also the community here can help you much more with it.

[rickjames 106]

You can try flashing pfSense on it,

https://lists.pfsense.org/pipermail/list/2014-February/005455.html

 

At least with an open source system there might be a chance that FreeBSD got drivers for that chipset.

Also the community here can help you much more with it.

 

​This ^

​500Mhz embedded without the use of crypto is going to have a hard time. Pfsense/freebsd 10 would at least have a chance.

[LazyLizard14 11]

Sorry guys but this is not leading anywhere....justifying a router by its price or comparing MHz (from various architectures) is not considered serious troubleshooting. Especially since most people here haven't heard about this router before.

Hardware offloading is working but doesn't make a difference if it is enabled or not in my case.

Screenshot taken during a "full speed" download at 7Mbit/s:

Still think the cpu is too busy?

[zhang888 1066]

Correct me if I am wrong but setting up the tunnel on the router should give me the same speeds as using the Eddie client on a windows machine if a) the same protocol settings and server used and   the router's cpu isn't maxed out. Right?

 

Wrong. Both of your statements are wrong, since you are not comparing apples to apples when taking a modern Intel CPU vs. 3 year old embedded router CPU.

 

If you are still not convinced, feel free to check the following link, or measure the same benchmark yourself.

 

http://wiki.openwrt.org/inbox/benchmark.openssl

 

Your router:

• Ubiquiti EdgeRouter Lite (e100) MIPS64 1000.0 0.9.8o w/o hw crypto 38823590 27464020 10354350 4249940 5946030 2134360 10099370 8791040 7765960 5.7 213.6 21.4 17.5

 

Vs another 70$ MIPS router:

• r42056 Qualcomm Atheros QCA9558 rev 0 TP-LINK Archer C7 MIPS 74Kc V5.0 358.80 1.0.1i 48201030 26489800 12468980 5741250 6431000 2313460 12628150 10951790 9591940 12.4 441.7 44.9 35.8

 

Vs another 70$ PPC based router:

 

• r42328 Freescale P1014 TP-Link TL-WDR4900 v1 PowerPC e500v2 99.99 1.0.1i 40339740 29738410 22216320 8258420 14423670 5276350 23153400 21997950 18524400 14.6 524.7 52.5 43.1

 

I marked the OpenSSL benchmark of AES-256 (which is used by AirVPN) in bold.

 

Considering the fact that the Archer C7 can max only 10Mbit, and the WDR4900 can do 15-18Mbit, the speed that you get is normal and expected.

And by the way, the CPU usage has nothing to do with that, we don't necessarily mean that your CPU maxed out it's 100% capacity and thats the reason for the speed.

 

 

Regards

[mage1982 15]

Sorry guys but this is not leading anywhere....justifying a router by its price or comparing MHz (from various architectures) is not considered serious troubleshooting. Especially since most people here haven't heard about this router before.

 

I sympathize with your frustration, I really do.

 

But try to see it from the other side: Since we can't get in to your router hardware and see what's going on, the alternatives are either suggesting things that are known to have been problems in many other cases (slow router, traffic shaping by ISP) or saying nothing at all.

 

None of us are experts on your particular hardware or the software that is running on it, we're just offering suggestions.

 

Perhaps the manufacturer has a support forum you could try.

 

 

Edit: Oh, it doesn't run custom closed source firmware? In that case I suppose some people here actually might know a thing or two about the software side of things.

Edited ... by mage1982

[LazyLizard14 11]

And by the way, the CPU usage has nothing to do with that, we don't necessarily mean that your CPU maxed out it's 100% capacity and thats the reason for the speed.

 

Thanks for clarifying this as here is the misunderstanding: I assumed when you talk of the CPU being too slow means it's constanly running under heavy load.

With the router supporting h/w crypto acceleration I not expected at all to run into this kind of issues. Interestingly the UBNT support is blaming OpenVPN architecture itself for the performance issues: click

Even with encryption disabled (or crypto offloaded) there is only a very limited performance gain to be expected!

The results on the openwrt website confirm the poor performance, but they were achieved with the initial firmware version about 2 years ago which truly did not support offloading as the comment "w/o hw crypto" states.

 

So I guess I not gonna investigate further about the wrong MTU size and packet fragmentation. Probably I gonna give pfSense a try.

 

Which router would you recommend for my connection (50 MBit/s cable)?

[zhang888 1066]

I hope the employee that made this comment, is not the one who is in charge of writing the software 

 

Yes the main bottleneck for OpenVPN appears to be the architecture so even if hardware crypto offload is supported, the performance gain would be limited (as indicated by the "no-encryption" result).

 

OpenVPN is not the one to blame here, OpenSSL is. Originally it was developed for x86 only, so you can't really hope for blazing fast speeds on other architectures.

Especially when you use extremely strong ciphers like AES-256.

 

I can only recommend you building your own firewall/router if you want to maximize the full capacity of your link.

That all depends on your budget, the number of users you wish to serve, and a little skills to build it.

 

Rangerly/Avoton to start with,

 

Or if you want to make a one time investment for upcoming 5+ years, you can get a Xeon CPU with a mini-ITX motherboard, it can do upto 500Mbit OpenVPN AES-256.

 

The most recommended consumer router would probably be WDR4900 (mpcC85xx 800Mhz version, NOT the ar71xx 720Mhz Chinese version) with OpenWRT, the only one that I am aware of, and according to benchmarks, that is capable of 18Mbit.

[SlyFox 10]

My ISP limits me to 30 Mbps. I bought the asus ac-56u for around $85. It's a 800mhz dual core and I can easily overclock it to 1.2ghz. It maxes out my ISP allowed speed. I tested the router at a friends house on his 50 Mbps line and it handled it fine. Running openvpn tomato on the router.

 

If my ISP was faster than 30-50 Mbps I would have went the route that zhang suggested but this router is good for my needs.

[Lee47 23]

People still use routers

 

Build one yourself and stick pfsense on it and unleash the bandwith and full options

 

https://airvpn.org/topic/11245-how-to-set-up-pfsense-21-for-airvpn/

[LazyLizard14 11]

Thanks for your answers. Indeed it looks like I haven't been wise enough to choose the proper router and / or fooled by UBNT, especially about the crypto h/w acceleration. The statement in their forums also seems strange that OpenVPN is the problem and not the encryption causing this slowdown.

Your suggestions gave a me a overview what to get to achieve reasonable speeds, tho it is not in my budget to spend a few hundred Euro in additional hardware. Well, I have a Windows 2012 server running with AD etc. but to connect it directly to the internet and set it up as NAT-Router isn't a good idea in terms of security. Running it behind the EdgeRouter means double-NAT; a bad choice either. Only FreeBSD would be worth a try although i doubt i can set it up...

As much as I like AirVPN and their terms of privacy and speed it probably wasn't the best choice for me as they only support OpenVPN (which isn't supported on Windows Phone at all). :-(

[flat4 79]

I hope the employee that made this comment, is not the one who is in charge of writing the software 

 

Yes the main bottleneck for OpenVPN appears to be the architecture so even if hardware crypto offload is supported, the performance gain would be limited (as indicated by the "no-encryption" result).

 

OpenVPN is not the one to blame here, OpenSSL is. Originally it was developed for x86 only, so you can't really hope for blazing fast speeds on other architectures.

Especially when you use extremely strong ciphers like AES-256.

 

I can only recommend you building your own firewall/router if you want to maximize the full capacity of your link.

That all depends on your budget, the number of users you wish to serve, and a little skills to build it.

 

Rangerly/Avoton to start with,

 

Or if you want to make a one time investment for upcoming 5+ years, you can get a Xeon CPU with a mini-ITX motherboard, it can do upto 500Mbit OpenVPN AES-256.

 

The most recommended consumer router would probably be WDR4900 (mpcC85xx 800Mhz version, NOT the ar71xx 720Mhz Chinese version) with OpenWRT, the only one that I am aware of, and according to benchmarks, that is capable of 18Mbit.

 

Nice board, great for a server

[go558a83nk 362]

sorry, but routers can do openvpn faster than what is stated in this thread.  My own Asus AC68 will do 35mbit/s, my max line speed.  I'm sure it could go faster if my ISP allowed it.

 

granted, it can't be an old, cheap router.  But, newer routers with ARM chips are certainly capable of very useful speeds.

[Khariz 109]

That actually one of the only consumer grade routers on the market that can hit 50mb. Most struggle to hit 15.

[go558a83nk 362]

the AC56 has the same CPU so it can. and the AC87 certainly can with its faster CPU.

 

several other brands with similar CPU can also do it.  but, one thing to note is that merlin firmware for Asus may have some openvpn optimizations.

 

also, I've seen some data that indicate that Astrill's applet for routers delivers impressive speeds when using their routerpro option.