DNS errors & making Bluetit and Ubuntu NetworkManager play nice with network lock

[postergus 0]

Hi there,

Long time hassle-free AirVPN user with a first troubleshooting request after switching from Eddie-UI to the Bluetit stack (love your work!)

Looks as though the two processes are fighting for `/etc/resolv.conf` and causing issues with DNS. NetworkManager will rewrite the file on a Wi-Fi network change, causing loss of connectivity as local DNS is disallowed by the network lock. Can be worked around by stopping the Bluetit service, toggling Wi-Fi, then re-enabling; but this is tedious to repeat if the network is at marginal signal strength.

Is there a way to configure NetworkManager not to mess with DNS? I think that would largely resolve the issue.
But, in an ideal world I would be able to have NetworkManager still manage the DNS if Bluetit is not active so that I can still operate normally on local networks without routing through AirVPN if I choose to. Seems unlikely to be simple, but worth asking.

Given that I am getting two warnings about DNS, I wonder if `systemd-resolved` could also be interfering and if there are other configuration steps I can take to ensure compatibility with it-

```
bluetit: WARNING: NetworkManager is running on this system and may interfere with DNS management and cause DNS leaks
bluetit: WARNING: systemd-resolved is running on this system and may interfere with DNS management and cause DNS leaks
```

I also wonder whether use of Goldcrest could avoid some of these problems. Personally I have not understood the need for the utility and have been interacting with Bluetit directly via systemctl and `/etc/airvpn/bluetit.rc`. As far as I can tell, Goldcrest just moves configuration stuff out of the `.rc` file into CLI args?

[Staff 10018]

@postergus

Hello!

Thank you for your great feedback!

Goldcrest offers the option to drive Bluetit with a fine grained access control (typically to any user in the group airvpn), instead of having to gain root privileges. It's a paramount security enhancement which is not underestimated by any serious UNIX administrator.

About the competition for DNS settings between Bluetit, systemd-resolved and Network Manager, there is no easy solution to implement in Bluetit, as each of the hundreds Linux distributions may work differently. Furthermore it would be probably dangerous (if possible at all) that Bluetit tried to block DNS operations by root processes or other daemons.

Just to say, systemd-resolved alone has several working modes: modes which bypass resolv.conf file and modes which don't. It's plausible that the best course of action is that each system administrator, according to her or his needs as well as system status, performs a fine-tuning.

Kind regards
 

[postergus 0]

Hrmm, are there any best-practises that folks familiar with these tools can recommend? I'm actually finding Bluetit somewhat unworkable in this mode with the Wi-Fi connection I'm on, have reverted to the old Eddie UI for now.

[OpenSourcerer 1442]

11 hours ago, postergus said:

Is there a way to configure NetworkManager not to mess with DNS?

Easiest way without breaking core network manager functionality is to work with its profiles. Create a dedicated profile with the DNS set to the servers in the specs. Switch manually between them before connection and after disconnection from AirVPN. NetworkManager will manage resolv.conf, and you can give Hummingbird/Goldcrest the -i switch.

If you're prepared to deal with divergent behavior, in /etc/NetworkManager/NetworkManager.conf, there in the [main] section, add the line:

dns=none

Restart NetworkManager.service. Then handle DNS by /etc/resolv.conf exclusively, manually. Unless systemd-resolved is also active, then simply stop and disable its service.

# systemctl stop systemd-resolved.service && systemctl disable systemd-resolved.service

.
 

[postergus 0]

Just wanted to share my full setup / config here as it might be useful to others (and it might be good to have other eyes across it )

I now have bluetit / AirVPN Suite playing very nicely with NetworkManager, including auto-reconnect when the network changes. This works well for me on Ubuntu 20.04 with AirVPN Suite 1.2.0.

The main aspects are disabling systemd-resolved and setting up a network script to stop & restart the VPN client when (non-tunnel) networks come back online.
 

#---- remove UFW package if installed- it conflicts

sudo apt-get remove -y ufw --purge


#---- disable systemd-resolved as it interferes with VPN-managed DNS

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved


#---- set networkmanager to use default DNS (not systemd-resolved)

sudo sed -i '/^\[main\]/a dns=default' /etc/NetworkManager/NetworkManager.conf
sudo rm /etc/resolv.conf
sudo systemctl restart NetworkManager


#---- add a network script to reconnect VPN on network change

sudo tee /etc/network/if-up.d/bluetit-reconnect > /dev/null <<END_DOC
#!/bin/env bash

if [[ $IFACE = tun* ]]
then
    logger "bluetit reconnect: $IFACE"
else
    logger "network $IFACE changed: reload bluetit"
    systemctl stop bluetit
    sleep 1
    systemctl start bluetit
fi
END_DOC
sudo chmod +x /etc/network/if-up.d/bluetit-reconnect

[postergus 0]

Upon reflection, it does occur to me that there will be network leakage for that 1 second during the bluetit daemon reconnection script.

Can anybody help with the shell commands needed to sever & reconnect the bluetit connection without disabling the network lock inbetween?

[Staff 10018]

On 11/21/2022 at 12:40 AM, postergus said:

Upon reflection, it does occur to me that there will be network leakage for that 1 second during the bluetit daemon reconnection script.

Can anybody help with the shell commands needed to sever & reconnect the bluetit connection without disabling the network lock inbetween?

Hello!

The "sleep 1" is unnecessary, you can delete it, as systemctl is synchronous. You can also order a restart to save time. Anyway you will still have a fraction of a second during which the ordinary, previous firewall rules are restored and Network Lock rules are not in place. At a first sight we see two alternative, possible solutions:

1. don't stop Bluetit but interact with it through Goldcrest. Apparently there's no reason whatsoever to restart Bluetit in your case: Bluetit re-reads network configuration at each session start, as you may notice from the log. You can have a goldcrest.rc runfile too, therefore you can configure Goldcrest to avoid any keyboard interaction. Make sure you have networklockpersist enabled in /etc/airvpn/bluetit.rc, so that the Network Lock rules will remain in place while Bluetit is not connected
2. keep blocking firewall rules in your system so no traffic will flow when Bluetit is not running: when Bluetit exits, it restores the blocking firewall rules. When it runs with networklockpersist, it enforces Network Lock. This second solution maintains the conceptual error to perform an unnecessary operation (restarting Bluetit) and requires total block rules by system default.

Kind regards
 

[neolost 0]

hello, but could you throw off the IP addresses of your data centers for tracing?

[postergus 0]

Thanks for this information! However it does not seem to be working as intended.

I have enabled the persistent network lock option in the bluetit config, which now looks like this (excluding bootserver, RSA & auth configs):

airconnectatboot        country
aircountry              SG
airipv6                 off
air6to4                 off
airblackcountrylist     US
networklockpersist      on

However, upon executing `systemctl stop bluetit` the network lock does seem to be disabled after all. Log output shows me:

Nov 22 13:22:22 beyongus bluetit: EVENT: DISCONNECTED
Nov 22 13:22:22 beyongus bluetit: Successfully restored DNS settings
Nov 22 13:22:22 beyongus bluetit: Network filter successfully restored
Nov 22 13:22:22 beyongus bluetit: OpenVPN3 connection thread finished
Nov 22 13:22:22 beyongus bluetit: Enabling persistent Network Filter and Lock
Nov 22 13:22:22 beyongus bluetit: Network filter and lock are using iptables-legacy
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_filter
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_nat
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_mangle
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_security
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_raw
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_filter
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_nat
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_mangle
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_security
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_raw
Nov 22 13:22:22 beyongus bluetit: Network filter successfully initialized
Nov 22 13:22:22 beyongus bluetit: Persistent network filter and lock successfully enabled
Nov 22 13:22:22 beyongus bluetit: Logging out AirVPN user postergus
Nov 22 13:22:22 beyongus bluetit: Enabling persistent network filter and lock
Nov 22 13:22:22 beyongus bluetit: Session network filter and lock are now disabled
Nov 22 13:22:22 beyongus bluetit: Network filter and lock are using iptables-legacy
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_filter
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_nat
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_mangle
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_security
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module iptable_raw
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_filter
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_nat
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_mangle
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_security
Nov 22 13:22:22 beyongus bluetit: Successfully loaded kernel module ip6table_raw
Nov 22 13:22:22 beyongus bluetit: Network filter successfully initialized
Nov 22 13:22:22 beyongus bluetit: Persistent network filter and lock successfully enabled
Nov 22 13:22:22 beyongus bluetit: AirVPN Manifest updater thread finished
Nov 22 13:22:22 beyongus bluetit: Session network filter and lock are now disabled

Any help with how to disable bluetit but keep the network lock active?

[Staff 10018]

@postergus

Hello!

The log shows that when the session network lock is disabled the persistent network lock is always enabled. When you shut down Bluetit, the persistent network lock is disabled as expected and intended. That's why we suggested as best solution that you do NOT shut down Bluetit. In your case, that's not only unnecessary, but even against what you want, we guess. Consider to manage Bluetit through Goldcrest for a proper usage for and a solution to your case. Please see also our previous message.

Kind regards
 

[postergus 0]

Hmm yeah I'm working through it now and I realise this is why I avoided Goldcrest in the beginning. Appreciate your support with this! 😄

In my network if-up.d I have tried the following variations:

    systemctl stop bluetit
    sleep 1
    systemctl start bluetit

Works, but loses the network lock (even if networklockpersist is on).

    goldcrest --reconnect

Broken. Causes the following loop in logs every 10 seconds-

Nov 25 10:51:20 beyongus bluetit: Server poll timeout, trying next remote entry...
Nov 25 10:51:20 beyongus bluetit: EVENT: RECONNECTING
Nov 25 10:51:20 beyongus bluetit: Contacting 185.200.117.133:443 via UDP
Nov 25 10:51:20 beyongus bluetit: EVENT: WAIT
Nov 25 10:51:20 beyongus bluetit: Connecting to [sg3.vpn.airdns.org]:443 (185.200.117.133) via UDPv4

Also requires the entire network to be taken down and brought back online in order to recover. (goldcrest --recover-network has no effect since it's unable to get a connection for the VPN node.) It's as if the DNS resolution is failing inbetween connections if triggered by Goldcrest?

(First question- will Goldcrest default to Bluetit configuration variables, if no configuration is passed to Goldcrest? Or do I have to duplicate my configuration?)

    systemctl stop bluetit
    systemctl start bluetit

Sometimes causes errors if your WiFi connection is unstable, due to rapid subsequent executions of the if-up.d script.

Nov 22 15:14:42 beyongus bluetit: EVENT: RECONNECTING
Nov 22 15:14:42 beyongus bluetit: Contacting 185.200.116.133:443 via UDP
Nov 22 15:14:42 beyongus bluetit: EVENT: WAIT
Nov 22 15:14:42 beyongus bluetit: Connecting to [sg3.vpn.airdns.org]:443 (185.200.116.133) via UDPv4
Nov 22 15:14:44 beyongus root: network wlp0s20f3 changed: reload bluetit
Nov 22 15:14:44 beyongus systemd[1]: bluetit.service: Succeeded.
Nov 22 15:14:44 beyongus bluetit: Received SIGTERM signal. Terminating Bluetit.
Nov 22 15:14:44 beyongus bluetit: Stopping OpenVPN3 connection thread
Nov 22 15:14:44 beyongus bluetit: Connection statistics updater thread finished
Nov 22 15:14:44 beyongus bluetit: net_route_del: 128.0.0.0/1 via 10.31.158.1 dev tun0 table 0 metric 0
Nov 22 15:14:44 beyongus bluetit: net_route_del: 0.0.0.0/1 via 10.31.158.1 dev tun0 table 0 metric 0
Nov 22 15:14:44 beyongus bluetit: net_addr_del: 10.31.158.16/24 dev tun0
Nov 22 15:14:44 beyongus bluetit: net_iface_mtu_set: mtu 1500 for tun0
Nov 22 15:14:44 beyongus bluetit: net_iface_up: set tun0 down
Nov 22 15:14:44 beyongus bluetit: net_route_del: 185.200.116.133/32 via 192.168.5.29 dev wlp0s20f3 table 0 metric 0
Nov 22 15:14:44 beyongus bluetit: sitnl_send: rtnl: generic error: No such process (-3)
Nov 22 15:14:44 beyongus bluetit: Error while executing NetlinkRoute4(add: 0) wlp0s20f3: -3
Nov 22 15:14:44 beyongus systemd[1]: bluetit.service: Found left-over process 8564 (bluetit) in control group while starting unit. Ignoring.
Nov 22 15:14:44 beyongus systemd[1]: bluetit.service: Will not start SendSIGKILL=no service of type KillMode=control-group or mixed while processes exist
Nov 22 15:14:44 beyongus systemd[1]: bluetit.service: Failed to run 'start' task: Device or resource busy
Nov 22 15:14:44 beyongus systemd[1]: bluetit.service: Failed with result 'resources'.
Nov 22 15:14:44 beyongus nm-dispatcher[12685]: Job for bluetit.service failed because of unavailable resources or another system error.

[Staff 10018]

@postergus

Hello!

Yes, reconnect is not bugged in the Suite, but the OpenVPN3-AirVPN library may mess up in the conditions of network down and up while a connection is ongoing. Same thing happens with OpenVPN 2.x, we see.  Now we understand why you chose to shut down the daemon completely.

Preferably disconnect first with "goldcrest --disconnect" and then send some goldcrest connect command: as far as we can see this procedure resolves the problem - and you will always have network lock enabled. Define your goldcrest.rc run control file to avoid keyboard interaction with Goldcrest:
https://airvpn.org/suite/readme/#goldcrest-configuration

goldcrest.rc directives and options will be  overridden by bluetit.rc system directives (in general, those directives which are not air string prefixed). If Goldcrest options or directives conflict with bluetit.rc non-air prefixed directives, Bluetit ones are enforced. For example if you tell Goldcrest to disable Network Lock when networklockpersist on is declared in bluetit.rc, Goldcrest will send the command to Bluetit, but the daemon will refuse to comply. The logic is that all users, according to root decisions, in airvpn group have privileges to change network and settings, except those system settings which must be exclusive competence of root.

You must specify in goldcrest.rc all the relevant for you connection related options (username and password for example are mandatory).

Kind regards