3 VPN connection using pfsense

[scout 0]

I am using pfsense and I have been maintaining 3 concurrent VPN connection for the last few months.  Always worked fine but starting a few days ago ( I can't exactly pinpoint it, maybe this Saturday?),I can't have 3 concurrent connections, only 2.  Its not a specific server that fail, its looks like whichever connection is going last is going to fail.

 

This is what I see on AirVPN client area :

 

 

Last attempted connection failed 43s ago.
Reason: Limit of 3 connections per account reached.

 

 

There are 3 sessions active on this account. Max 3 concurrent sessions allowed.

  Server Location Connected since Total Traffic Current Speed Visible in internet
with IP (Exit) Coming from
(Your IP) Force
disconnection Download Upload Download Upload 1 Naos United Kingdom, Manchester 5m 7s ago 26 kB 55 kB 7 B/s 135 B/s 84.39.117.57

xxx

 

Disconnect now 2 Keid Netherlands, Amsterdam 51s ago 6 kB 5 kB 0 B/s 0 B/s 95.211.138.33

xxx

 

Disconnect now 3 Alkaid United States, Chicago, Illinois 5m 10s ago 214 kB 55 kB 150 B/s 132 B/s 46.21.154.83

xxx

 

Disconnect now

 

 

But in pfsense, I only have two connections valid, logs are below.  If I force disconnect the connection on AriVPN and restart it from pfsense, I get the same errors.

 

OpenVPN options in pfsense :

 

remote-cert-tls server;tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA;keysize 256;auth SHA1;key-method 2;key-direction 1;comp-lzo no;verb 3;explicit-exit-notify 5; remote 149.255.33.154 443

 

 

 pfsense logs :

 

Sep 29 17:20:21

openvpn[36947]: SIGTERM[soft,exit-with-notification] received, process exiting

 

 

Sep 29 17:20:21

openvpn[36947]: SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)

 

 

Sep 29 17:20:16

openvpn[36947]: SIGTERM received, sending exit notification to peer

 

Sep 29 17:20:16

openvpn[36947]: AUTH: Received control message: AUTH_FAILED

 

Sep 29 17:20:16

openvpn[36947]: SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)

 

Sep 29 17:20:13

openvpn[36947]: [server] Peer Connection Initiated with [AF_INET]95.211.138.7:443

 

Sep 29 17:20:13

openvpn[36947]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 4096 bit RSA

 

Sep 29 17:20:13

openvpn[36947]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication

 

Sep 29 17:20:13

openvpn[36947]: Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key

 

Sep 29 17:20:13

openvpn[36947]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication

 

Sep 29 17:20:13

openvpn[36947]: Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key

 

Sep 29 17:20:06

openvpn[36947]: VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=server, emailAddress=info@airvpn.org

 

Sep 29 17:20:06

openvpn[36947]: VERIFY EKU OK

 

Sep 29 17:20:06

openvpn[36947]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication

 

Sep 29 17:20:06

openvpn[36947]: Validating certificate extended key usage

 

Sep 29 17:20:06

openvpn[36947]: VERIFY KU OK

 

Sep 29 17:20:06

openvpn[36947]: ++ Certificate has key usage 00a0, expects 00a0

 

Sep 29 17:20:06

openvpn[36947]: Validating certificate key usage

 

Sep 29 17:20:06

openvpn[36947]: VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org

 

Sep 29 17:20:05

openvpn[36947]: TLS: Initial packet from [AF_INET]95.211.138.7:443, sid=1ec16427 4b134f99

 

Sep 29 17:20:05

openvpn[36947]: UDPv4 link remote: [AF_INET]95.211.138.7:443

 

Sep 29 17:20:05

openvpn[36947]: UDPv4 link local (bound): [AF_INET]XXXXXXXXX

 

Sep 29 17:20:05

openvpn[36876]: Socket Buffers: R=[42080->65536] S=[57344->65536]

 

Sep 29 17:20:05

openvpn[36876]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication

 

Sep 29 17:20:05

openvpn[36876]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication

 

Sep 29 17:20:05

openvpn[36876]: Control Channel Authentication: using '/var/etc/openvpn/client4.tls-auth' as a OpenVPN static key file

 

Sep 29 17:20:05

openvpn[36876]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts

 

Sep 29 17:20:05

openvpn[36876]: MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client4.sock

 

Sep 29 17:20:05

openvpn[36876]: OpenVPN 2.3.2 amd64-portbld-freebsd8.3 [sSL (OpenSSL)] [LZO] [eurephia] [MH] [iPv6] built on Mar 27 2014

 

 

 

 

 

Any ideas where to start looking to fix this?  Thanks

[scout 0]

I disabled them all, rebooted and started them one at time and it just worked.  Not sure what it is but I am up and running now.

 

Thans

[securvark 16]

I disabled them all, rebooted and started them one at time and it just worked.  Not sure what it is but I am up and running now.

 

Thans

Hey, I realize this is an older thread, I hope you're still around.

 

Could you elaborate on how you created 3 VPN connections?

 

I read it should be possible with PfSense to create them and even bundle them so PfSense does load balancing over them.

 

The reason I would like to do it this way is that openvpn is single threaded and at ~100mbit it caps a CPU core in the virtual machine. If I can give it 4 cores with 3 openvpn instances, I should be able to get more bandwidth and spread the load over multiple cores.

 

Thanks!

[SirJohnEh 6]

Just create 3 separate client profiles for VPN where each one connects to a different server.  To minimize problems, I find it best to configure each connection to use a different port.  By doing that, each connection gets a unique 10.x address (one on 10.4.x, one on 10.6.x and one on 10.30.x), making routing much easier.  Once you have all three profiles created and connected then just setup your NAT rules for each, create a routing group to load balance across each connection, then setup the policy based routing in your LAN firewall rules.

 

I've been running a similar setup since using AirVPN.  My box has no problems sustaining my full 60Mbs on any one link, but I like having multiple connections in a round robin especially during the work day.  If there's problems with one of my connections then pfsense just drops that server and routes thru the others seamlessly.  Then when the bad link is healthy again, pfsense just adds it back to the group and starts using it again.

 

The other nice thing I do with it is to switch where I route thru at night for things like netflix.  With 3 connections always active, I can just easily switch the gateway for my streaming device and point it to the US link instead of Canada and bam I'm instantly on US netflix, etc.

[securvark 16]

Thanks for the help.

 

I can create the extra VPN's, am I correct in assuming I need to choose TCP instead of UDP or doesn't that matter? Either way they come up and work fine.

 

Do I follow the guide here on setting up AirVPN for PfSense with respect to the Interface Assigning, and gateway config for each one?

 

On the OpenVPN client config page, do I need to configure Tunnel settings like the IP4 Tunnel Network?

 

The moment I create the routing group and try setting it as a gateway, all VPN tunnels go down and the logs aren't clear on what is wrong.

 

This weekend I'll be able to try some more. Really appreciate it if you could guide me through this.

Thanks in advance!

[SirJohnEh 6]

Not 100% sure, but it sounds like you're trying to set the pfsense box's default gateway to the vpn tunnel.  That should work as long as the tunnels are already up and active, but will fail miserably if your ISP connection goes down temporarily or if you rebooted the pfsense box, lost power and it rebooted, etc.  Your vpn can't be the default gateway for your pfsense box because the vpn can't connect unless the pfsense box can first go thru your ISP to connect the tunnels.  I mean you could get around that with a bunch of rules, etc.  I guess it depends how badly you need to hide every packet from your ISP.  Me?  I leave my ISP as the default gateway so that the pfsense box can always reconnect the tunnels on reboot, etc.  But the only thing pfsense does is connect to airvpn and check for updates so I don't care about hiding that traffic from my ISP (and of course you can't hide the actual vpn tunnel connections anyway).  I think I also set it up to send email alerts when there are problems, but again I don't care if that traffic goes thru my ISP.  But if you have a need to ensure every packet that leaves your network is encrypted then you've got a little extra work ahead of you.  It's certainly possible, but will take a little extra effort.

 

I think I looked at the guide when setting up my box, but I already had experience with OpenVPN so I didn't use it for much if I did read it over so I can't say much about the guide.

 

UDP is preferred over TCP as it will provide a faster connection, all things being equal.

 

Leave the settings like tunnel network, etc. blank.  The server will provide that info on a connection (refer to the setup guide for details like that).

 

So when I create my routing group, I don't set it as the default gateway for the box.  Instead, I add a LAN fw rule that sets the gateway for all devices to the routing group.  Again, what this does is force every device on your LAN to route thru the vpn, but the pfsense box itself does not -- pfsense will route only thru your ISP.  For me, this is acceptable.

[securvark 16]

Thanks, appreciate you're taking the time to explain.

 

I am not setting my VPN gateway as the default gateway, that's bad practise and using policy based routing is the preferred way as far as I'm concerned.

 

I'll try to go through it later today or tomorrow, and I'll record what I'm doing. Hopefully it will be clear to you (or another kind soul willing to help ) to see what I'm doing wrong and kick me in the right direction.

[securvark 16]

Ha! It's working .

 

3 connections being used simultaneously. Not by the same connections of course, but 3 individual downloads from different sites will use the different VPN tunnels.

Did notice that even after resetting/restarting services, a reboot is sometimes required to bring everything back up.

 

Pretty awesome nonetheless!

 

Thanks again for your help mate! Appreciate it.

[SirJohnEh 6]

Glad you got it working!

[hammerman 3]

i always use 2 vpn connections . . . (have to save one for when i'm not home.)
i set up as per pfsense_fan's instructions and just repeat for the other vpn connection.

it's simple but it seems to work ok. 

could you elaborate on the part . . . "create a routing group to load balance across each connection, then setup the policy based routing in your LAN firewall rules."

is it better to have policy based routing when you have sufficient ethernet ports available?

[SirJohnEh 6]

So there's only two ways pfsense is going to route traffic thru the vpn for you:

 

Either 1) you've set the default gateway of the pfsense box to be your vpn tunnel or 2) you configure policy based routing thru firewall rules.

 

#1 just isn't a good idea (see my reasons above).  If circumstances warrant the need for everything, including pfsense, to go thru vpn then it can be done, but it's a hassle and is prone to issues (and mostly user errors).

 

#2 is called policy based routing because you're explicitly setting the gateway based on (firewall) rules, that's all.

 

 

There's my policy based routing and that's all there is to it.  So everything on the LAN is set to use my VPNOnly group as its gateway.  With that, everything has to go thru the vpn.  If all connections in the group are disconnected for whatever reason then the rule is auto disabled by pfsense and since there's no other rules below it, all LAN traffic is blocked -- basically creating a kill switch to force every device on my LAN to either use the vpn or they get no internet access.

 

 

And then there's my routing group.  My two Canada links are both on Tier 1 so they'll be used round robin.  And that's all there is to it.

 

My third connection is either not connected (I usually use it on my phone/tablet/laptop when on the road) or if it is connected then it's usually a USA link (sometimes a Euro link, if needed) and what I'll do is enable the rule above that is shown as disabled.  By turning on that rule, it's another policy based rule that then forces my Roku to route thru whichever link I choose (USA in the example shown) and that's how I set the country for Netflix.  If that rule is disabled then the Roku just uses the same rule as everyone else (round robins between Canada links).  If I turn on the rule then I also set the gateway to a specific location (usually USA; make sure you've enabled that OpenVPN client connection and it's up before enabling the rule) to change the Netflix catalog based on whichever catalog has the movie we want to watch.

 

I only have 2 NICs in my pfsense box -- 1 WAN, 1 LAN.  I use multiple AirVPN links on pfsense in a round robin for one main reason only: redundancy.  All my LAN traffic goes thru vpn and I also work at home.  So during the day I can't afford to have my Internet go down.  With multiple vpn connections in a round robin, if one of the vpn servers goes bad for a little while, I don't even notice (other than the email pfsense sends me) as the other connection(s) just seamlessly takes on all traffic.  It's just a way I set things up to reduce the likelihood of an outage during my day.  Air servers sometimes go down -- maintenance, other user(s) taxing it to its limits, DDoS, whatever.  The odds of all Air servers I'm connected to going down at the same time?  Very, very unlikely, which means during my work day I'm never likely to be without internet access (thru the vpn).  The other reason I like to keep multiple connections is for the Netflix region changing, etc.  When multiple tunnels are connected, switching which one I route my Roku thru is quick and painless.  Just enable the rule to force the traffic down whichever tunnel I want and that's it.  So it makes my Netflix region switching a little more convenient.  But the main reason is connection redundancy during my work day.  The number of NICs is irrelevant for this anyway, assuming you have only one ISP connection because ultimately everything has to funnel thru your ISP anyways.  What you can do with multiple LAN NICs is segregate your traffic based on network source.  So direct each LAN NIC down its own vpn tunnel, etc.  But you could even do that with just one LAN NIC and setup VLANs, for example, to achieve the same kind of effect (assuming you have switches capable of supporting VLANs, etc.).

 

The sky's the limit with how you can set all this up, really.  In reality, I've simplified my examples above just to show the basics for policy based routing.  If I pasted in my full LAN fw rules you'd see other entries where I've setup schedules for some rules that changes the routing during the overnight hours so that my one vm on my LAN routes thru a Euro link and I do all my torrents for the day during overnight on that link (because though my ISP, like most in Canada, does UBB billing, they do not count traffic towards my cap between 2am-8am nightly so I queue up all my torrents during the day and let them go nuts during that 6 hour window every night).  I choose to route that bt traffic thru a Euro link mostly as a courtesy to AirVPN.  Even though I'll never see a DMCA notice torrenting thru Canada or the US, I'm guessing AirVPN still does so I figure I'll try to save them some paper work if I can.  Yeah, the link to Europe is higher latency, but I still get my full speed and the latency is irrelevant to me since I'm in bed.   At 8am, pfsense automatically changes the routing rules back so I'm routing thru my low latency Canada links during the day.  And with that, life is good!

[securvark 16]

SirJohnEh, are you also running a OpenVPN server to connect from the outside to home?

 

I've been able to set it up and connect with my Android. I can reach the local network but 1) DNS is not working and 2) I loose internet access.

 

When connected to my HomeVPN, I would want the clients' internet to be routed through VPN and I want PfSense to direct this traffic to the AirVPN routing group.

 

This is getting confusing for me because I can't seem to figure out which firewall zone (tab: wan, lan, openvpn) I need to configure things. I've been going over the firewall logs and packet dumps and I have no idea where things are going wrong.

 

I noticed on the assign interface, the homeVPN is listed and not yet assigned. Do I need to do this and use it to create firewall rules with policy based routing? Any idea on which zone I need to be?

 

Thanks for the help!

[SirJohnEh 6]

Yes, I also have a vpn server to connect from the outside into the LAN.  Harder to troubleshoot as we're getting into more and more variables.

 

If you're forcing the outside clients to route all traffic thru the vpn then, yes, you'll need to create fw rules to policy route it thru the air vpn group and you'll also need NAT rules to get the internet traffic from the vpn server properly NAT'd.

 

For DNS, are you assigning the user the DNS server from your pfsense box?  If not, then to access another DNS server, you're forcing it to go thru the vpn and access it from your box's internet connection.  If your user has no internet when connected then they won't have any DNS access unless you assign them the DNS server from your pfsense box.  Even with internet access, a cellular provider's DNS server probably won't allow connections from your pfsense box.  Be sure you're assigning your vpn clients the pfsense dns server.

 

So it's important to realize that the OpenVPN interface in pfsense applies to all OpenVPN connections -- client or server.  Because of this, I have no rules on the OpenVPN interface, blocking all traffic there.  I have port forwards for Air and so I don't want to apply those port forwards on my server interface.  And I allow my vpn clients to access anything, anywhere on my LAN -- and I definitely don't want that kind of access to traffic coming from my Air connections.  So this is why there is a separate interface created and active for all vpn connections (client or server).  On each (Air) client interface, I setup my port forwards from Air to allow that traffic thru as desired.  On the vpn server interface, I have the basic allow all traffic rule, with the gateway set to my VpnOnly group.  With that, incoming vpn users route all traffic thru my air tunnels.  And on the "OpenVPN" interface there are no rules.

 

Hopefully this helps, but you really need to be careful with this kind of stuff.  A misconfig here could open up your network in ways you don't intend (and certainly don't want).