OpenVPN Authenticate/Decrypt errors

[Whisperer 2]

A few days ago I updated my OpenWRT router. After working out the little kinks I ran into, I am now left with one major problem. Every time I download something using bittorrent (I download a LOT of Ubuntu ISO's obviously) I get maybe 1/3 of the speed and my whole log is spammed by " AEAD Decrypt error: bad packet ID (may be a replay) " messages. Since I didn't really change much about the network infrastructure or the VPN configuration, I was wondering why I get this now. And how to fix it. I read about the mssfix thing, I believe the staff recommended setting would be 1400.

But why does this happen in the first place? Is something adding padding to the frames? Can somebody here help me investigate and understand it better?

I asked the same question on the OpenWRT forum, to approach this from two sides. I really want to get a handle on this and get my stable connection back.

[OpenSourcerer 1442]

The primary contact would be OpenWRT, of course.
If you don't get this with a single HTTP download, I'd say OpenWRT tries to do QoS now, some "mild" packet reordering for better throughput and with prioritization. If it was indeed an upgrade of OpenWRT, I can imagine something like this.

[Whisperer 2]

I had not even realized I ought to test it using other protocols as well. So I went back to the Ubuntu site and downloaded the ISO directly from their server. Using a multi-threaded download I can push through to reach higher speeds, but the log is indeed being spammed by these messages, also with a standard http download.

I did the no-fragments ping test that I've seen mentioned in different places and the highest I can get is 1472. Which is normal for a 1500 MTU I believe.

So, what's next?

[OpenSourcerer 1442]

It started with an OpenWRT upgrade. Therefore I'd really focus the search there. I'm not too familiar with it, I'm afraid, but others here know a thing or two. Let's see if someone turns up.

[Whisperer 2]

Thanks for replying so fast.

By a lucky accident, the older build was still on the router in the backup  partition. Turns out the issue is there as well, but it was drowned out by log entries that were caused by a bug at the time. But after testing it with a single connection I can see the same flood.

That said, I think the issue may be on AirVPN's side after all. If all I'm reading about this issue is related to packetsize, and AirVPN's servers operate on a 1500 MTU, then someone from the staff please explain to me what this means:

I checkend the log from when openvpn was establishing the connection. Somewhere at the bottom I run a little script to switch out the ISP's DNS with the AirVPN's DNS to prevent any accidental leak while the tunnel is up. After that script it said

tun0 1500 1553 10.x.x.x 255.255.255.0 init

I get the IP address of my endpoint/client. I also get the mask. But what about the 1500 1533? Is that a packetsize? and isn't 1533 too high?

Then I checked the rest of the log and saw this:

PUSH: Received control message: 'PUSH_REPLY,comp-lzo no,redirect-gateway  def1 bypass-dhcp,dhcp-option DNS 10.xx.xx.1,route-gateway 10.xx.xx.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.xx.xx.xx 255.255.255.0,peer-id 5,cipher AES-256-GCM'

OPTIONS IMPORT: timers and/or timeouts modified
OPTIONS IMPORT: compression parms modified
OPTIONS IMPORT: --ifconfig/up options modified
OPTIONS IMPORT: route options modified
OPTIONS IMPORT: route-related options modified
OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
OPTIONS IMPORT: peer-id set
OPTIONS IMPORT: adjusting link_mtu to 1625
OPTIONS IMPORT: data channel crypto options modified

I can recognize the server's IP for DNS and gateway, my router's IP and mask and some options that are also in the client config and thus are redundant.

Then OpenVPN processes those values:
timers and/or timeouts modified -> no idea
compression parms modified -> that would be "comp_lzo no"
--ifconfig/up options modified -> setting the client's IP
route options modified & route-related options modified -> could be several things
--ip-win32 and/or --dhcp-option options modified -> Probably the DNS being pushed
peer-id set -> check
adjusting link_mtu to 1625 -> WAIT, WHAT?!

Is there an invisible option between "peer-id 5" and "cipher AES-256-GCM"

Isn't that 1625 not the actual issue here? I'm happy it is larger rather than smaller, but what effect does this have and why can't I send any packet higher than 1472 (probably because my network is set at 1500)?

Thinking out loud here (why stop now?), but there is a modem/router in front of my router/openvpn client. It is quite likely that it is set at an MTU of 1500 as well. So even if my openvpn router sends out a packet with a 1625 size, my ISP's modem will break that up in two pieces and send it on. But then again, AirVPN's endpoint is never going to be the next hop in my path anyway, and most of the internet operates IPv4 at an MTU of 1500.

So, if you (or anybody) can get one of those super tech-savvy staffmembers I hear so much about to tell me what I am not seeing here, I would appreciate it.

By the way, giga, the post you made in this thread was informative. I found someone on Reddit taking it up a notch though:
IP size : 20 bytes, UPD size : 8 bytes, VPN overhead : 41 bytes, VPN "options" : 4 bytes (according to the auther of openvpn apparently)
Surprisingly, that brings me full circle. A link_mtu of 1625 -20 -8 -41 -4 = 1552

Remember that 1500 1553 I noticed first? 1552, 1553. Pretty darn close.

Anyway... if someone could give me a definitive answer as to what I can set to avoid this problem (likely caused by fragmentation) I would greatly appreciate it.

[OpenSourcerer 1442]

5 minutes ago, Whisperer said:

That said, I think the issue may be on AirVPN's side after all.

Haha, no. Unless tens or hundreds of other clients have exact the same issue but choose to stay mum, which does not reflect my experience with AirVPN community at all. On the slightest sight of a problem #metoo posts start to appear pretty much all over the place. It'd be fun if it wasn't looking so helplessly.

The error states the packet ID is wrong, so technically OpenVPN dropped all the bad packets. That's why your throughput dissipates: Packets don't arrive at the end, the application. It can have a multitude of reasons, including packet fragmentation, as you try to troubleshoot, and high packet latency, which depends on the internet connection largely, but also general latency to AirVPN server. And of course the router's settings can impale VPN throughput simply because many of them somehow try to QoS (read: make "better") TCP and UDP connection, mosty UDP, though, because TCP "behaves".

Anyway, I think you first need to really understand the four options tun-mtu, link-mtu, mssfix and fragment and know what exactly they are changing, especially why the documentation states that link-mtu and tun-mtu should not be touched unless you really know what you're doing. Should I explain it to you or do you not want me to spoil you the joy of your own research? It's really only logical.

[Whisperer 2]

Okay, maybe I spoke too soon. I apologize. As you say, understanding those 4 options is not easy.

You should, however, probably explain them, since I spent the last hour experimenting by bringing my VPN down, setting an option and bringing it back up. AirVPN may start to think it's a DoS attack. 😉

This is what I know so far...

mtu-test gives me: local->remote=[1525,1525] remote->local=[1525,1525]

fragment will cause the tunnel to be established with two warnings:

WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1562', remote='link-mtu 1558'
WARNING: 'mtu-dynamic' is present in local config but missing in remote config, local='mtu-dynamic'

But no data will go through. Instead, it logs 6 times

FRAG_IN error flags=0xfa2a187b: FRAG_TEST not implemented

Then breaks the tunnel and re-establishes it with the same result.

mssfix seems to have no mention in the log or have any effect whatsoever.

The way I understand it is that mssfix is supposed to tell any stateful (TCP) connections to keep their packages at a certain size. If they don't listen, fragment should catch them and split them up into the right size, adding 4 bytes extra overhead.

There's mtu-disc, which breaks OpenVPN:

--mtu-disc is not supported on this OS
Exiting due to fatal error

Which leaves the twp options I'm not supposed to touch.

Neither have extensive descriptions, but both seem to be the only way to force my client to operate differently. The question is, what could I set them to?

EDIT:
I still think that if those packets break somewhere along the line, it could be my ISP's modem. I do believe if I ask really nicely, they will give me a newer model that at least leases an IPv6 address to my router. If it also exposes an IPv6 on the Internet, I could connect that way and maybe not have this issue anymore. I haven't actually set the modem into bridge-mode, but I did turn off pretty much every functionality and just pass everything through to the router.

[Whisperer 2]

Just to exclude it, I bit the bullet and asked the ISP to send me a newer modem. I was reluctant to do so previously since the one I have is a Cisco with lots configurable options (and I do like my control), but I suppose it was about time.

The plus side is that the new model doesn't just give my OpenWRT an IPv6 address, it gives me a whole public address space. And it doesn't even have a public IPv4 address anymore. So I will connect using IPv6 and see if the MTU discovery improves then. For the time being I'm working on a TCP tunnel. Which gives me about 2/3 of my previous max and less simultaneous connections. But at least my log is clear.

[OpenSourcerer 1442]

3 hours ago, Whisperer said:

So I will connect using IPv6 and see if the MTU discovery improves then.

And while you're at it, delete tun-mtu and link-mtu from your config and simply use the default as noted in the manpage. It might have a reason why it works best for most users.

You mentioned Cisco and a lot of customization options. Could that customization of yours be the reason?

[Whisperer 2]

I don't have tun-mtu and link-mtu in my config, because I don't know how I would set them. All I have in there are the options from the ovpn file, exactly as they are offered from the generator. Otherwise I wouldn't have come here asking for help. It was only after nobody could give me a clear 1-2-3 of how to fix this issue that I felt I should at least experiment.

And I pretty much disabled all the options in the Cisco, did everything short of turning it into a bridge. But I like knowing I have the option. 🙂

[OpenSourcerer 1442]

Ah, I see. I thought you already did your experiments.

[Whisperer 2]

As you already indicated, the subject matter is not exactly easy to understand. I tried to for several hours to figure out what I could do myself before changing anything, then chose to ask for help. But like most users in this world of instant-gratification, I couldn't wait too long before at least trying some stuff that I could easily revert. Did I have link-mtu in there at some point after opening this thread? Yup. Got a warning. Didn't fix the issue. So I removed it. One change at a time.

Still hoping somebody can provide a functional work-around. Even if the new router/modem fixes all, it bugs the heck out of me I wasn't able to fix this.

That said, "there is no way to fix this without entering identical settings server-side" is also an answer. As long as it is correct and there truly is no solution.

[drum 6]

I also "suffer" from this issue. I have tried multiple ways to solved it... reset router, new windows installation, configuring different MTUs, opened AirVPn ticket, talk to my ISP technicians.
Nothing sorted out the problem. But on the bright side i noticed that it doesn't affect my download/upload speed, as I almost get my 200/200mbit line saturated using AirVPN. So I learned to just ignore those errors messages.
Of course I'd like to see them away, but as long as it seems they are not affecting my system (at least noticeable to me) I'm fine.
I know this is not a solution, just wanted to share my experience.

[Whisperer 2]

Finally, a kindred spirit! 😉

Lucky you, I get 2/3rd of my max. But the new modem just arrived, so let's see what happens. At the very least, communicating over the Internet with IPv6 puts me back with the cool kids again. Worst case I have to figure out how to survive in a time with COVID and without Internet...

200mbit? You must be streaming Netflix in 4K all day long to saturate that. On multiple devices. 🙂

I really wonder how much ISPs worry if their high bandwidth subscriptions are gonna have an expiration-date as measures opposing the free flow of information increase. I'm sure you know what I mean by that.

[OpenSourcerer 1442]

@samb @Whisperer
Hm. What's your country, ISP, subscribed throughput and type of internet connection (DSL, DOCSIS/Cable, Fibre, GSM/LTE,…)?

[Whisperer 2]

Turning off IPv4 in your router is a very reliable method of finding out which websites don't have IPv4 yet. 🙂

Surprisingly, most websites I had open actually still work. I did not expect that.

I actually got 256MBit when plugged in directly to the modem just now. Using AirVPN I never got above 60. It's like a whole new world. Now to configure this thing.

You're asking for a lot of identifiable info on a public space.

[OpenSourcerer 1442]

11 minutes ago, Whisperer said:

You're asking for a lot of identifiable info on a public space.

All of this info is too vague to pinpoint anyone, but if you insist, you can also message me this info. The only reason I ask is to compare for similarities.
My line would be: "I'm in Germany on a 250/40 VDSL2 Supervectoring line with Telekom." I will buy you a beer if you manage to identify me with only this info.
 

[Whisperer 2]

Well, if somebody really wanted to, they could start with the provider. If I did not share the provider, they may be able to figure it out through the connection type. At this point, they can also determine which geographical area they service. Then they contact said provider and ask who just today received a new modem. Done! 🙂

In other words, I may just edit a few posts above.

Strangely enough, AirVPN's dynamic hostname does not resolve. Even after I enabled IPv4. And AirVPN.org resolves to its IPv4 and ::1, that's a fun one too. I smell something weird. Gonna see if downloading a config with IP will at least get me reconnected.

EDIT: And the resolved IP is an IPv4. That's kind of silly.
EDIT2: "RESOLVE: Cannot resolve host address: <IPADDRESS> (Name does not resolve)"
Back to the drawing board, somethng's weird with my router.
EDIT3: I can only connect when using IPv4. And apparently I am leaking IPv6 now, as my system is connected to the tunnel but happily bypassing it using its IPv6 address. So much work...

Okay, so by default my PC gets an Internet routable IP, which causes it to go directly out there. If I stop that, no more leak. On the downside, speed over my VPN is now back to 47Mbps, a 200Mbps decrease..

[OpenSourcerer 1442]

40 minutes ago, Whisperer said:

Well, if somebody really wanted to, they could start with the provider. If I did not share the provider, they may be able to figure it out through the connection type. At this point, they can also determine which geographical area they service. Then they contact said provider and ask who just today received a new modem. Done! 🙂

Paranoia. For the last few years I have been proposing that any kind of paranoia is to be destroyed at once. Know why? Because it's in the way of rational, logical thinking. It makes people fear the most unlikely events, the ominous unknown, what might happen. And it always stems from a simple lack of knowledge, or the inability or unwillingness to think about it for a moment.

Would someone really, in a capitalistic, economy-driven world, spend money and manpower (read: time) just to track you down, individually? Only because you use a VPN and posted here? Would someone really contact your ISP and ask these things? These are very, VERY, unlikely events.
But okay, say "they" did that for whatever idiotic reason and "they" actually got to your address or something. What then? Nothing. "They" can't sue you because you did nothing wrong, "they" can't spy on you because you use a VPN, and if "they" invest even more manpower and money to actually follow you around and document what you do, well, they're complete idiots economy-wise, but rejoice – you're the center of their lives apparently. Do The Truman, carry on.

1 hour ago, Whisperer said:

Strangely enough, AirVPN's dynamic hostname does not resolve.

Sorry, what hostnames do you mean by that?

[Whisperer 2]

Maybe Ubuntu doesn't like it that I download their torrent all the time? 🙂

You know, the FQDN hostnames, like de.vpn.airvpn.org.
But since it gave me the same error when using an IP address there was something strange gong on. It connects as long as I connect using IPv4, but as soon as I set the protocol to IPv6, OpenVPN bugs out. Even though it has both an IPv6 IP and an IPv6 DNS available.

Since the Config generator also provides me with an IPv4 address for the VPN server even when I select I wish to use UDP6 to connect, I'm highly confused. If you select to generate a config for IPv6, wouldn't you expect an IPv6 IP Address for your server? How can you establish an IPv6 tunnel when you connect over IPv4?

[OpenSourcerer 1442]

1 hour ago, Whisperer said:

You know, the FQDN hostnames, like de.vpn.airvpn.org.

The schema is explained in a FAQ entry: 

1 hour ago, Whisperer said:

Since the Config generator also provides me with an IPv4 address for the VPN server even when I select I wish to use UDP6 to connect, I'm highly confused. If you select to generate a config for IPv6, wouldn't you expect an IPv6 IP Address for your server? How can you establish an IPv6 tunnel when you connect over IPv4?

Have you ticked Advanced Mode and selected "Connect with IP layer: IPv6"? By default the generator creates configs for both v4 and v6 and connects you with v4.

[Whisperer 2]

I did, actually, and even when I selected to connect to IPv6 it still gave me the DDNS name for IPv4 in the OVPN file. I guess it's a bug. But I will try it once more, just to be certain.

Nope. Even if I select an IPv6 entry AND exit point, I still get the IPv4 DDNS name. I guess I found a bug. But that's fine, now that I know I can manually change it by putting IPv6 in the name.

Of course, the trouble did not stop there. Now the server won't give my OpenVPN client its IPv6 address in the PUSH message. Or the DNS. So I still end up with an IPv4 tunnel, even though the encapsulation is running on IPv6.

All this just to get rid of an error message. 🙂

[Whisperer 2]

Phew, solved that issue as well. Bug in the router's OS wasn't sending all the options in the OVPN file to OpenVPN, so it never asked for IPv6 info.

Bug report about config generator to AirVPN: Check
Bug report about OpenVPN issue to OpenWRT: Check

Please, for the sake of my sanity, let it work now! 🙂

[Whisperer 2]

Well, I'm giving up. I spent the last 13 hours, clear through the night trying to get it all working again, and the speed just gets slower and slower. By now I'm grateful if I make it to 10Mbit out of my 250 line. A whopping 1Mbyte/s download. And judging from the responses to support requests and their participation on the forum, hoping that a staff member helps me out is just waiting for my inevitable end.

[OpenSourcerer 1442]

7 hours ago, Whisperer said:

I did, actually, and even when I selected to connect to IPv6 it still gave me the DDNS name for IPv4 in the OVPN file. I guess it's a bug. But I will try it once more, just to be certain.

Oh wow, I can confirm this! Ticking Resolved hosts explicitly shows v4 despite the fact v6 was chosen if a country/continent/earth is selected. It works with individual servers, though.
Really looks like a little bug. Thank you for reporting it.