[How To FIX] pfSense and multiple VPN tunnels

securvark

If you are using multiple OpenVPN tunnels from the same machine (router, firewall, whatever) you are probably not getting your multiple tunnels. Here's how to check and fix it along the way. Mind you, this only applies if you are opening multiple VPN tunnels from the same machine.

I've asked about this here and there's replies there that lead to possible solutions which are just too complicated for simply guy like me. This is what I've come up with after some mailing around with AirVPN's Tech Support.

I am using pfSense so my instructions are based on that. If you are using another router, firewall or whatever else that's out there, you're on your own. My tips here should provide you with a way to check and fix it so read along.

Check your OpenVPN tunnels. In pfSense, simply open Status / OpenVPN to get the overview:

AirVPN Belgium UDP4 up Mon May 21 11:47:02 2018 192.168.1.10:56152 10.6.0.27   194.187.251.162:80   13 KiB / 28 KiB
AirVPN Germany UDP4 up Mon May 21 11:46:46 2018 192.168.1.10:55677 10.4.27.248 185.189.112.26:443   17 KiB / 34 KiB
AirVPN Sweden UDP4  up Mon May 21 11:46:45 2018 192.168.1.10:41699 10.30.1.2   62.102.148.144:1194  23 KiB / 14 KiB
AirVPN Swits UDP4   up Mon May 21 11:46:45 2018 192.168.1.10:38526 10.30.0.214 185.156.175.34:2018  41 KiB / 38 KiB
AirVPN GB UDP4      up Mon May 21 11:46:47 2018 192.168.1.10:24082 10.30.1.95  185.103.96.132:41185 25 KiB / 19 KiB

Check your routing table. In pfSense, Diagnostic, Routes. Ignore IPv6.

Routing tables Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS         re0
10.4.0.0/16        10.4.0.1           UGS      ovpnc2
10.4.0.1           link#8             UH       ovpnc2
10.4.27.248        link#8             UHS         lo0
10.6.0.0/16        10.6.0.1           UGS      ovpnc1
10.6.0.1           link#7             UH       ovpnc1
10.6.0.27          link#7             UHS         lo0
172.16.1.0/24      link#2             U           re1
172.16.1.10        link#2             UHS         lo0
10.30.0.0/16       10.30.0.1          UGS      ovpnc3
10.30.0.1          link#9             UH       ovpnc3
10.30.0.214        link#10            UHS         lo0
10.30.1.2          link#9             UHS         lo0
10.30.1.95         link#11            UHS         lo0
127.0.0.1          link#4             UH          lo0
192.168.1.0/24     link#1             U           re0
192.168.1.254      link#1             UHS         lo0
194.187.251.154/32 192.168.1.1        UGS         re0

Oke, if you know anything about subnetting and gateways, you should immediately see what is going on here. For those that don't ... I can't (don't want to) explain how subnetting works, but just understand this: IP's and subnets are like streets and numbers with zipcodes. A bunch of streets with numbers are in an area that share the same zipcode. The zipcode is like a gateway address; all mail for the addresses that share a zipcode are sent there where it gets divided to streets and numbers. Disclaimer: I know this is a terrible analogy and its actually not correct - I know! It works for what I'm trying to explain and if you wanna know how it works, Google it! .

Now, look at the virtual IP's. Above they are:

10.6.0.27
10.4.27.248
10.30.1.2
10.30.0.214
10.30.1.95

I've placed them below each other so you can see that 3 of them start with 10.30. The way AirVPN has configured their servers is that any IP that has the same second octet (the 2nd number in the IP address) is in the same subnet and shares the gateway. Now look at the routing table, you'll see these:

10.4.0.0/16        10.4.0.1           UGS      ovpnc2
10.4.0.1           link#8             UH       ovpnc2
10.4.27.248        link#8             UHS         lo0
10.6.0.0/16        10.6.0.1           UGS      ovpnc1
10.6.0.1           link#7             UH       ovpnc1
10.6.0.27          link#7             UHS         lo0
10.30.0.0/16       10.30.0.1          UGS      ovpnc3
10.30.0.1          link#9             UH       ovpnc3
10.30.0.214        link#10            UHS         lo0
10.30.1.2          link#9             UHS         lo0
10.30.1.95         link#11            UHS         lo0

What you see here is that all addresses in:
10.4.0.0/16 go to gateway 10.4.0.1 on interface ovpnc2
10.6.0.0/16 go to gateway 10.6.0.1 on interface ovpnc1
10.30.0.0/16 go to gateway 10.30.0.1 on interface ovpnc3

So, where are the other 2 OpenVPN tunnels, link#10 and link#11? They have interface lo0, not ovpnc4 and ovpnc5 as you would expect.

And that is what is going wrong: 10.30.1.2 on link#9 and 10.30.1.95 on link#11 are in the range 10.30.0.0/16 and share the gateway on link#9, ovpnc3. All their traffic gets sent to ovpnc3, their connections to their servers are unused and all their traffic gets sent to the server behind ovpnc3 on link#9.

Now, how do we fix this? it's actually very simple:
Well, the way AirVPN hands out their virtual IP's works as follows:
Generation 1 servers (those that do not support TLS or IPv6) get their range based on the port you're connecting to:
port 53 10.8.0.0/16
port 80 10.6.0.0/16
port 443 10.4.0.0/16
ports 1194 and higher 10.30.0.0/16

Generation 2 servers all get 10.4.0.0/16 regardless of their port number.

So now its simple to fix this. Set each connection to a different port above and make one connection to a Gen2 server. This is the result:

BE_Gen1_80 UDP4 	up 	Fri May 25 18:04:43 2018 	192.168.1.10:15895 	10.6.0.128 	194.187.251.90:80 	99.60 MiB / 5.42 GiB
ALL_Gen2_443 UDP4 	up 	Fri May 25 18:04:44 2018 	192.168.1.10:24410 	10.14.0.147	91.207.57.114:443 	160.08 MiB / 4.85 GiB
DE_Gen1_53 UDP4 	up 	Fri May 25 18:04:45 2018 	192.168.1.10:41624 	10.8.0.80 	185.189.112.18:53 	1.35 GiB / 6.19 GiB
SE_Gen1_1194 UDP4 	up 	Fri May 25 18:04:51 2018 	192.168.1.10:29541 	10.30.0.109 	62.102.148.149:1194 	832.47 MiB / 1.80 GiB
NL_Gen1_443 UDP4 	up 	Fri May 25 18:04:54 2018 	192.168.1.10:42418 	10.4.52.230 	109.232.227.137:443 	117.24 MiB / 6.31 GiB

Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default		192.168.1.1	UGS	6896589	1500	re0	
10.4.0.0/16	10.4.0.1	UGS	0	1500	ovpnc5	
10.4.0.1	link#11		UH	0	1500	ovpnc5	
10.4.52.230	link#11		UHS	238780	16384	lo0	
10.6.0.0/16	10.6.0.1	UGS	0	1500	ovpnc1	
10.6.0.1	link#7		UH	0	1500	ovpnc1	
10.6.0.128	link#7		UHS	238774	16384	lo0	
10.8.0.0/16	10.8.0.1	UGS	0	1500	ovpnc3	
10.8.0.1	link#9		UH	0	1500	ovpnc3	
10.8.0.80	link#9		UHS	238764	16384	lo0	
10.14.0.0/16	10.14.0.1	UGS	0	1500	ovpnc2	
10.14.0.1	link#8		UH	0	1500	ovpnc2	
10.14.0.147	link#8		UHS	238892	16384	lo0	
10.30.0.0/16	10.30.0.1	UGS	0	1500	ovpnc4	
10.30.0.1	link#10		UH	0	1500	ovpnc4	
10.30.0.109	link#10		UHS	238762	16384	lo0	
127.0.0.1	link#4		UH	37972	16384	lo0	
172.16.123.45	link#2		UHS	0	16384	lo0	
172.16.123.45/32link#2		U	0	1500	re1	
192.168.1.0/24	link#1		U	150536	1500	re0	
192.168.1.10    link#1		UHS	0	16384	lo0

As you can see all my connections are in unique subnets and in the routing table you can see that each tunnel has its own interface and gateway.

Now if you are connecting to country FQDN's to let AirVPN DNS resolve you to a "random" server, you may be thinking, how do I prevent getting a Generation 2 server when I want to connect to a Generation 1 on a specific port to get that unique subnet and gateway? Well, the anwer is, don't use FQDN's. Connect your vpn tunnels to a specific IP address and use the 'remote' directive in the custom options box with all the other IP's you want to connect to. You can get the IP's from the Client Area.

Hope this helps! Questions or tips and corrections? Please post!

go558a83nk

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

securvark

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I currently only use "don't pull" but I tested both and each one separately, and also the topology. Nothing changes what the server pushes to the client.

go558a83nk

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I currently only use "don't pull" but I tested both and each one separately, and also the topology. Nothing changes what the server pushes to the client.

Much of my pfsense setup is following the excellent guide that's here on the forums. In that don't pull and don't add/remove are selected and therefore you don't have problems of subnets overlapping. And perhaps there are some other settings that affect this too that I don't recall. Anyway, I've never had problems running multiple clients.

What the server pushes doesn't change but it's that pfsense ignores it.

securvark

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I currently only use "don't pull" but I tested both and each one separately, and also the topology. Nothing changes what the server pushes to the client.

Much of my pfsense setup is following the excellent guide that's here on the forums. In that don't pull and don't add/remove are selected and therefore you don't have problems of subnets overlapping. And perhaps there are some other settings that affect this too that I don't recall. Anyway, I've never had problems running multiple clients.

What the server pushes doesn't change but it's that pfsense ignores it.

The guide is indeed excellent, and I use it for my base config too.

Have you read this?

The issue I describe here is not new, and Air VPN support is aware of this issue. Couple of years ago they moved from a /30 topology to a /16. I am not making this up.

If you are using multiple VPN client connections from the same machine (pfsense or router), please take your time to check your routing table. Go to pfSense, Status, OpenVPN, and record the Virtual Address of each connection. Do any of them share the same 2nd octet, like multiple in the 10.4 or 10.30 range? Look at your routing table, go to pfSense, Diagnostics, Routes. Check that you see all your OpenVPN clients under the "netif" collumn, or just paste both here and let me have a look.

To which ports did you setup your connections?

go558a83nk

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I currently only use "don't pull" but I tested both and each one separately, and also the topology. Nothing changes what the server pushes to the client.

Much of my pfsense setup is following the excellent guide that's here on the forums. In that don't pull and don't add/remove are selected and therefore you don't have problems of subnets overlapping. And perhaps there are some other settings that affect this too that I don't recall. Anyway, I've never had problems running multiple clients.

What the server pushes doesn't change but it's that pfsense ignores it.
The guide is indeed excellent, and I use it for my base config too.

Have you read this?

The issue I describe here is not new, and Air VPN support is aware of this issue. Couple of years ago they moved from a /30 topology to a /16. I am not making this up.

If you are using multiple VPN client connections from the same machine (pfsense or router), please take your time to check your routing table. Go to pfSense, Status, OpenVPN, and record the Virtual Address of each connection. Do any of them share the same 2nd octet, like multiple in the 10.4 or 10.30 range? Look at your routing table, go to pfSense, Diagnostics, Routes. Check that you see all your OpenVPN clients under the "netif" collumn, or just paste both here and let me have a look.

To which ports did you setup your connections?

I've definitely run multiple openvpn clients where the subnets overlapped. yet, I've never had trouble with getting the traffic I wanted through the tunnel I wanted via NAT and firewall rules.

securvark

I suggest you re-read my original post. Obviously you didn't understand the part about the subnets and gateways.

NaDre

...

I've definitely run multiple openvpn clients where the subnets overlapped. yet, I've never had trouble with getting the traffic I wanted through the tunnel I wanted via NAT and firewall rules.

I suggest you re-read my original post. Obviously you didn't understand the part about the subnets and gateways.

You may be mistaken there. In IPTABLES (the Linux low-level firewall) or the BSD low-level firewalls (PF, IPFW) the rules refer to specific interfaces. So even if these interfaces have the same IP address, they can still be distinguished.

The one case where having the same IP address on two interfaces is an issue is if you want to bind a program to one of them. The "bind" call that programs issue does not allow the actual interface to be specified - just the address.

EDIT: For the sake of completeness, it is possible for a program to bind to a specific interface using a "setsockopt" call to set the "SO_BINDTODEVICE" option. But few programs do this and allow you to set an interface. See:

http://man7.org/linux/man-pages/man2/getsockopt.2.html

http://man7.org/linux/man-pages/man7/socket.7.html

securvark

...

I've definitely run multiple openvpn clients where the subnets overlapped. yet, I've never had trouble with getting the traffic I wanted through the tunnel I wanted via NAT and firewall rules.
>I suggest you re-read my original post. Obviously you didn't understand the part about the subnets and gateways.

You may be mistaken there. In IPTABLES (the Linux low-level firewall) or the BSD low-level firewalls (PF, IPFW) the rules refer to specific interfaces. So even if these interfaces have the same IP address, they can still be distinguished.

The one case where having the same IP address on two interfaces is an issue is if you want to bind a program to one of them. The "bind" call that programs issue does not allow the actual interface to be specified - just the address.

oke, I'm keen to learn, if I got it wrong, please explain. Are you saying the traffic from 10.30.0.214 and 10.30.1.95 is not sent to 10.30.0.1, but is kept on their respective links 10 and 11 and end up on the correct VPN public IP address without an actual gateway defined? They have their interface set to lo0, not an ovpnc#. See below:

10.30.0.0/16       10.30.0.1          UGS      ovpnc3
10.30.0.1          link#9             UH       ovpnc3
10.30.0.214        link#10            UHS         lo0
10.30.1.2          link#9             UHS         lo0
10.30.1.95         link#11            UHS         lo0

If you look at that routing table (snip from my initial post), you'll see the netif for the 10.30.0.0/16 is on ovpnc3 which has its gateway on link#9. The other 2, 10.30.0.214 and 10.30.1.95 on link 10 and 11 respectively, fall in the same range 10.30.0.0/16. As far as I'm concerned, all traffic from those interfaces to addresses outside their own subnet, are sent to the gateway set for that subnet, which is 10.30.0.1 on link 9. All that traffic from 10.30.0.0/16 will end up on ovpnc3, I do not see how traffic to the internet from 10.30.0.214 or 10.30.1.95 can end up anywhere else but the VPN server behind 10.30.0.1 (62.102.148.144:1194 from the table above).

I've contacted support and I've posted about this on the pfsense forums, and both acknowledge the problem, maybe they got it wrong too? I don't know ...

Really, I'm not trying to be cocky or anything, I'm happy to accept I got it wrong, I just don't see how.

NaDre

..
oke, I'm keen to learn, if I got it wrong, please explain. ...

You did not get it wrong. The routing table you first described was certainly not going to work as you wanted.

Are you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I took this to mean that go558a83nk was suggesting that you should not let OpenVPN set up the routing table, since the result of that was broken. And instead you would have to set the routing table up correctly yourself, probably using "--up" scripts.

But this would require a considerable amount of understanding. And if you can avoid the issue by choosing server ports in a certain way, as you did, then that was probably a very good idea.

My response was just because you seemed to think that go558a83nk did not understand about sub-nets. But the fact that go558a83nk had said "via NAT and firewall rules" suggested to me that she/he may have some deeper level of expertise.

go558a83nk

I'm no expert. There's way too much I don't understand.

The OP may be on to something but I also figured if this were an inevitable problem it would be discussed a lot more in pfsense openvpn topics.

Here's my routing table with two clients up and running. The subnets aren't the same since one of them is to a tls-crypt server now.

securvark

..
oke, I'm keen to learn, if I got it wrong, please explain. ...

You did not get it wrong. The routing table you first described was certainly not going to work as you wanted.

Thanks for confirming.

re you using the options in the openvpn client setup "don't pull routes" and "don't add/remove routes"?

I took this to mean that go558a83nk was suggesting that you should not let OpenVPN set up the routing table, since the result of that was broken. And instead you would have to set the routing table up correctly yourself, probably using "--up" scripts.

But this would require a considerable amount of understanding. And if you can avoid the issue by choosing server ports in a certain way, as you did, then that was probably a very good idea.

My response was just because you seemed to think that go558a83nk did not understand about sub-nets. But the fact that go558a83nk had said "via NAT and firewall rules" suggested to me that she/he may have some deeper level of expertise.

You're right. I was too blunt there and jumped to conclusions.

Just for information for you all, "don't pull routes" and "don't add/remove routes" don't do anything. You can tick em both, restart OpenVPN and nothing changes in the routing tables. If the server is configured to not allow client overrides, these won't do anything. Same with network topology setting and some others.

I'm no expert. There's way too much I don't understand.

The OP may be on to something but I also figured if this were an inevitable problem it would be discussed a lot more in pfsense openvpn topics.

Here's my routing table with two clients up and running. The subnets aren't the same since one of them is to a tls-crypt server now.

First, apologies for my previous reply.

Setup a few more connections on the same ports and see what happens . But it's not inevitable as the first post in this thread fixes it .

I posted about this on the pfSense forums to which a Netgate employee replied. You can find it here:

https://forum.netgate.com/topic/131155/multiple-clients-range-overlap

There's few other threads here on the forums too, if you care to search for it.

Last but not least, I've been in contact with AirVPN tech support and they acknowledge the issue too.

I've ran for months with overlapping subnets, thinking it was working just fine. The thing is, it looks fine. If you follow the pfsense setup guide here on the forum and choose the /30 topology, people (at least I did) assume you're getting /30 so you won't have overlapping subnets. Moreover, the VPN's come up and appear to be working. You'll see counters go up under Status/OpenVPN and if you have the network graphs on your Dashboard, you'll see the graphs drawing traffic to all tunnels. Makes me wonder how many people here have this problem on their router, without actually realizing there's a problem with their connection.

go558a83nk

So, I did a little test. I connected to two different servers but at the same port. So, both had a 10.30.0.x virtual address. Sure enough, in the routing table of the web GUI only "ovpnc 1" showed under the netif column even though two clients were running according to "ifconfig" at the command line.

However, when I looked at what my exit IP was according to web sites, machines that were supposed to use server "A" were reported as having server A's exit IP. And machines supposed to use the server "B" were reported as having the exit IP of server B.

Could this just a bug in routing table display?

securvark

So, I did a little test. I connected to two different servers but at the same port. So, both had a 10.30.0.x virtual address. Sure enough, in the routing table of the web GUI only "ovpnc 1" showed under the netif column even though two clients were running according to "ifconfig" at the command line.

However, when I looked at what my exit IP was according to web sites, machines that were supposed to use server "A" were reported as having server A's exit IP. And machines supposed to use the server "B" were reported as having the exit IP of server B.

Could this just a bug in routing table display?

I really don't know.

I see the same behavior, but ipleak.net page sometimes loads only partially on a server that shares a gateway with a real link. On the airvpn.org client area, I have 2 connections now, one to NL and another to SE. NL is the "real" one, but SE is counting traffic too. It more or less matches what my pfSense graph is showing, about 1/5th of what the NL is showing.

But here's a funny one:

The SE server IP is detected, but under IP details it is showing the NL server and geolocation.

So I have no idea what is really going on, this one is beyond me. I know one thing though: it's not working correctly

Judas4all

So now its simple to fix this. Set each connection to a different port above and make one connection to a Gen2 server. This is the result:

BE_Gen1_80 UDP4 	up 	Fri May 25 18:04:43 2018 	192.168.1.10:15895 	10.6.0.128 	194.187.251.90:80 	99.60 MiB / 5.42 GiB
ALL_Gen2_443 UDP4 	up 	Fri May 25 18:04:44 2018 	192.168.1.10:24410 	10.14.0.147	91.207.57.114:443 	160.08 MiB / 4.85 GiB
DE_Gen1_53 UDP4 	up 	Fri May 25 18:04:45 2018 	192.168.1.10:41624 	10.8.0.80 	185.189.112.18:53 	1.35 GiB / 6.19 GiB
SE_Gen1_1194 UDP4 	up 	Fri May 25 18:04:51 2018 	192.168.1.10:29541 	10.30.0.109 	62.102.148.149:1194 	832.47 MiB / 1.80 GiB
NL_Gen1_443 UDP4 	up 	Fri May 25 18:04:54 2018 	192.168.1.10:42418 	10.4.52.230 	109.232.227.137:443 	117.24 MiB / 6.31 GiB

Hi, sorry, I'm not sure what you want me to do ?

securvark

Hi, sorry, I'm not sure what you want me to do ?

Depends.

Have you read about the routing table and checked to see if you an issue?

lordlukan

I have been doing the same with my pfsense routers (2 routers linked via openvpn tunnel). Use different ports for different servers. Also, TCP/UDP use different subnets, so there are plenty to go around for 5 concurrent connections

dIecbasC

This works for up to 4 on separate subnets https://nguvu.org/pfsense/pfsense-multi-vpn-wan/ and could easily be extended to 5 with the OPs solution to use the Gen2 server as connection 5. It could be worth running a wireshark capture on the dual subnets to see whats going on, my foolish guess is that traffic exits the correct interface but always returns on the initial gateway. Im guessing through... I might find time this weekend to verify. Im curious.

[How To FIX] pfSense and multiple VPN tunnels

Recommended Posts

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Share this post

Link to post

Join the conversation