ANSWERED Using OpenVPN config, unable to get DNS push working

[thewolfman8326 0]

Short: Using OpenVPN configuration, edited config file to run update-systemd-resolved, still not seeing AirVPN dns when checking on ipleak.net.

Longer:
Arch Linux
OpenVPN version 2.5.0


My OVPN config

[me@hostname Kruger]$ cat AirVPN_US-Chicago-Illinois_Kruger_UDP-443.ovpn 
# --------------------------------------------------------
# Air VPN | https://airvpn.org | Sunday 3rd of January 2021 06:48:09 PM
# OpenVPN Client Configuration
# AirVPN_US-Chicago-Illinois_Kruger_UDP-443
# --------------------------------------------------------

client
dev tun
remote 68.235.35.123 443
resolv-retry infinite
nobind
persist-key
persist-tun
auth-nocache
route-delay 5
verb 3
explicit-exit-notify 5
push-peer-info
setenv UV_IPV6 yes
ca "ca.crt"
cert "user.crt"
key "user.key"
remote-cert-tls server
comp-lzo no
data-ciphers AES-256-GCM:AES-256-CBC:AES-192-GCM:AES-192-CBC:AES-128-GCM:AES-128-CBC
data-ciphers-fallback AES-256-CBC
proto udp
tls-auth "ta.key" 1

script-security 2
setenv PATH /usr/bin
up /etc/openvpn/scripts/update-systemd-resolved          
down /etc/openvpn/scripts/update-systemd-resolved
down-pre

dhcp-option DOMAIN-ROUTE

Output of terminal when connecting with OpenVPN

[me@hostname Kruger]$ sudo openvpn AirVPN_US-Chicago-Illinois_Kruger_UDP-443.ovpn 
[sudo] password for me: 
2021-01-16 17:23:13 WARNING: file 'user.key' is group or others accessible
2021-01-16 17:23:13 OpenVPN 2.5.0 [git:makepkg/a73072d8f780e888+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Nov  6 2020
2021-01-16 17:23:13 library versions: OpenSSL 1.1.1i  8 Dec 2020, LZO 2.10
2021-01-16 17:23:13 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2021-01-16 17:23:13 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2021-01-16 17:23:13 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
2021-01-16 17:23:13 TCP/UDP: Preserving recently used remote address: [AF_INET]68.235.35.123:443
2021-01-16 17:23:13 Socket Buffers: R=[212992->212992] S=[212992->212992]
2021-01-16 17:23:13 UDP link local: (not bound)
2021-01-16 17:23:13 UDP link remote: [AF_INET]68.235.35.123:443
2021-01-16 17:23:13 TLS: Initial packet from [AF_INET]68.235.35.123:443, sid=ba9f17a5 79159312
2021-01-16 17:23:13 net_route_v4_best_gw query: dst 0.0.0.0
2021-01-16 17:23:13 net_route_v4_best_gw result: via 10.0.0.1 dev enp0s31f6
2021-01-16 17:23:13 VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org
2021-01-16 17:23:13 VERIFY KU OK
2021-01-16 17:23:13 Validating certificate extended key usage
2021-01-16 17:23:13 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2021-01-16 17:23:13 VERIFY EKU OK
2021-01-16 17:23:13 VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=Kruger, emailAddress=info@airvpn.org
2021-01-16 17:23:13 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_CHACHA20_POLY1305_SHA256, 4096 bit RSA
2021-01-16 17:23:13 [Kruger] Peer Connection Initiated with [AF_INET]68.235.35.123:443
2021-01-16 17:23:14 SENT CONTROL [Kruger]: 'PUSH_REQUEST' (status=1)
2021-01-16 17:23:14 PUSH: Received control message: 'PUSH_REPLY,comp-lzo no,redirect-gateway ipv6 def1 bypass-dhcp,dhcp-option DNS 10.20.136.1,dhcp-option DNS6 fde6:7a:7d20:1088::1,tun-ipv6,route-gateway 10.20.136.1,topology subnet,ping 10,ping-restart 60,ifconfig-ipv6 fde6:7a:7d20:1088::105b/64 fde6:7a:7d20:1088::1,ifconfig 10.20.136.93 255.255.255.0,peer-id 2,cipher AES-256-GCM'
2021-01-16 17:23:14 OPTIONS IMPORT: timers and/or timeouts modified
2021-01-16 17:23:14 OPTIONS IMPORT: compression parms modified
2021-01-16 17:23:14 OPTIONS IMPORT: --ifconfig/up options modified
2021-01-16 17:23:14 OPTIONS IMPORT: route options modified
2021-01-16 17:23:14 OPTIONS IMPORT: route-related options modified
2021-01-16 17:23:14 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
2021-01-16 17:23:14 OPTIONS IMPORT: peer-id set
2021-01-16 17:23:14 OPTIONS IMPORT: adjusting link_mtu to 1625
2021-01-16 17:23:14 OPTIONS IMPORT: data channel crypto options modified
2021-01-16 17:23:14 Data Channel: using negotiated cipher 'AES-256-GCM'
2021-01-16 17:23:14 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2021-01-16 17:23:14 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2021-01-16 17:23:14 net_route_v4_best_gw query: dst 0.0.0.0
2021-01-16 17:23:14 net_route_v4_best_gw result: via 10.0.0.1 dev enp0s31f6
2021-01-16 17:23:14 ROUTE_GATEWAY 10.0.0.1/255.255.255.0 IFACE=enp0s31f6 HWADDR=4c:cc:6a:6b:95:89
2021-01-16 17:23:14 GDG6: remote_host_ipv6=n/a
2021-01-16 17:23:14 net_route_v6_best_gw query: dst ::
2021-01-16 17:23:14 net_route_v6_best_gw result: via fe80::10:18ff:fe77:60b2 dev enp0s31f6
2021-01-16 17:23:14 ROUTE6_GATEWAY fe80::10:18ff:fe77:60b2 IFACE=enp0s31f6
2021-01-16 17:23:14 TUN/TAP device tun0 opened
2021-01-16 17:23:14 net_iface_mtu_set: mtu 1500 for tun0
2021-01-16 17:23:14 net_iface_up: set tun0 up
2021-01-16 17:23:14 net_addr_v4_add: 10.20.136.93/24 dev tun0
2021-01-16 17:23:14 net_iface_mtu_set: mtu 1500 for tun0
2021-01-16 17:23:14 net_iface_up: set tun0 up
2021-01-16 17:23:14 net_addr_v6_add: fde6:7a:7d20:1088::105b/64 dev tun0
2021-01-16 17:23:14 /etc/openvpn/scripts/update-systemd-resolved tun0 1500 1553 10.20.136.93 255.255.255.0 init
<14>Jan 16 17:23:14 update-systemd-resolved: Link 'tun0' coming up
<14>Jan 16 17:23:14 update-systemd-resolved: Adding DNS Routed Domain DOMAIN-ROUTE
<14>Jan 16 17:23:14 update-systemd-resolved: Adding IPv4 DNS Server 10.20.136.1
<14>Jan 16 17:23:14 update-systemd-resolved: Adding IPv6 DNS Server fde6:7a:7d20:1088::1
<14>Jan 16 17:23:14 update-systemd-resolved: SetLinkDNS(4 2 2 4 10 20 136 1 10 16 253 230 0 122 125 32 16 136 0 0 0 0 0 0 0 1)
<14>Jan 16 17:23:14 update-systemd-resolved: SetLinkDomains(4 1 DOMAIN-ROUTE true)
2021-01-16 17:23:20 net_route_v4_add: 68.235.35.123/32 via 10.0.0.1 dev [NULL] table 0 metric -1
2021-01-16 17:23:20 net_route_v4_add: 0.0.0.0/1 via 10.20.136.1 dev [NULL] table 0 metric -1
2021-01-16 17:23:20 net_route_v4_add: 128.0.0.0/1 via 10.20.136.1 dev [NULL] table 0 metric -1
2021-01-16 17:23:20 add_route_ipv6(::/3 -> fde6:7a:7d20:1088::1 metric -1) dev tun0
2021-01-16 17:23:20 net_route_v6_add: ::/3 via :: dev tun0 table 0 metric -1
2021-01-16 17:23:20 add_route_ipv6(2000::/4 -> fde6:7a:7d20:1088::1 metric -1) dev tun0
2021-01-16 17:23:20 net_route_v6_add: 2000::/4 via :: dev tun0 table 0 metric -1
2021-01-16 17:23:20 add_route_ipv6(3000::/4 -> fde6:7a:7d20:1088::1 metric -1) dev tun0
2021-01-16 17:23:20 net_route_v6_add: 3000::/4 via :: dev tun0 table 0 metric -1
2021-01-16 17:23:20 add_route_ipv6(fc00::/7 -> fde6:7a:7d20:1088::1 metric -1) dev tun0
2021-01-16 17:23:20 net_route_v6_add: fc00::/7 via :: dev tun0 table 0 metric -1
2021-01-16 17:23:20 Initialization Sequence Completed

Output of resolv.conf (does not change after connecting)

[me@hostname ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 1.1.1.1
nameserver 8.8.8.8

Output of resolvectl status after connecting to VPN

Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported           
    resolv.conf mode: foreign                                                  
  Current DNS Server: 1.1.1.1                                                  
         DNS Servers: 1.1.1.1 8.8.8.8                                          
Fallback DNS Servers: 1.1.1.1 9.9.9.10 8.8.8.8 2606:4700:4700::1111 2620:fe::10
                      2001:4860:4860::8888                                     

Link 2 (enp0s31f6)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                   
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1                                                     
       DNS Servers: 1.1.1.1 8.8.8.8                                             

Link 3 (wlp0s20f0u12)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                   
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.20.136.1                                                 
       DNS Servers: 10.20.136.1 fde6:7a:7d20:1088::1                            
        DNS Domain: ~DOMAIN-ROUTE  

After all this, the DNS from ipleak.net does not show it as a valid AirVPN DNS server. I was once able to get it working once by manually editing resolv.conf to have 10.0.4.1 (which I believe is supposed to be universal across all servers, correct?), but I have not been able to replicate this again.

For clarification, I am using Network Manager for my regular connection, but am NOT using network-manager-openvpn; I use openvpn directly to connect. I'm assuming this has something to do with network manager but not sure exactly how all the inner workings connect.

Thank you for any help
  Edited ... by thewolfman8326

[OpenSourcerer 1441]

20 hours ago, thewolfman8326 said:

After all this, the DNS from ipleak.net does not show it as a valid AirVPN DNS server. I was once able to get it working once by manually editing resolv.conf to have 10.0.4.1 (which I believe is supposed to be universal across all servers, correct?), but I have not been able to replicate this again.

Maybe because the server is 10.4.0.1 instead?
 

20 hours ago, thewolfman8326 said:

For clarification, I am using Network Manager for my regular connection, but am NOT using network-manager-openvpn; I use openvpn directly to connect. I'm assuming this has something to do with network manager but not sure exactly how all the inner workings connect.

Can't you create a separate NetworkManager profile, then change the profiles using nmcli before connection? Use nmcli connection show to show the list of connections and find out their UUIDs, copy the UUID of the AirDNS connection, then use up /usr/bin/nmcli connection up UUID in the config to change to the OpenVPN connection. Revert with the same command, using the UUID of the usually used connection profile instead.

[thewolfman8326 0]

On 1/17/2021 at 2:33 PM, OpenSourcerer said:

Maybe because the server is 10.4.0.1 instead?

Sorry, that was a typo, I did mean 10.4.0.1! Updating original post to clarify (when I did it, ipleak showed it as an AirVPN exit node)
 

On 1/17/2021 at 2:33 PM, OpenSourcerer said:

Can't you create a separate NetworkManager profile, then change the profiles using nmcli before connection? Use nmcli connection show to show the list of connections and find out their UUIDs, copy the UUID of the AirDNS connection, then use up /usr/bin/nmcli connection up UUID in the config to change to the OpenVPN connection. Revert with the same command, using the UUID of the usually used connection profile instead.

I think slight misunderstanding on one of our parts here; since I don't have the AirVPN connection setup through networkmanager, nmcli connection show doesn't list any UUID for it; only the one for my wired connection. Or did you mean create a connection through that first?

However I did notice that after running the openvpn command, my resolvctl status info says my tun0 "current DNS" updates, but my wired "Current DNS" does not (see original post). When modifying my wired connection to ONLY have 10.4.0.1, it seems to work, as that gets set in my "Current DNS" for the wired connection. Somehow Firefox is taking that DNS instead of the tun0 DNS maybe?

I'm unfortunately in a rush right now and can't keep troubleshooting, but I'm going to try and make a second wired connection through networkmanager that only has 10.4.0.1 as the sole DNS server option, and try that, and will report back.

Thank you for your response!
 

[OpenSourcerer 1441]

12 hours ago, thewolfman8326 said:

Or did you mean create a connection through that first?

Yes.
 

12 hours ago, thewolfman8326 said:

Somehow Firefox is taking that DNS instead of the tun0 DNS maybe?

It's all about speed. If Cloudflare DNS outside the tunnel is faster than AirDNS inside the tunnel, I can imagine Cloudflare gets the request.

[Staff 10014]

@thewolfman8326

Hello!

Strange, as from your reports it seems that update-systemd-resolved is no more sufficient to accept effectively DNS push for certain DNS configurations in Linux. We assume that you have systemd-resolved running in some "on link" mode, or anyway bypassing /etc/resolv.conf, is this correct or not? What is your distribution?

In theory, now Bluetit and Hummingbird (they are in the AirVPN Suite software) should handle properly all the numerous "DNS methods" available in Linux to date, even the Windows-like, disgraceful ones which have been adopted lately by some distributions. Can you please test and check whether Bluetit (or Hummingbird) are able to set, during an OpenVPN session, properly DNS in your Linux box AND prevent DNS leaks, or not?

See also:
https://airvpn.org/forums/topic/48833-linux-airvpn-suite-100-released/

(make sure to read notes about systemd-resolved as well)

Kind regards
 

[thewolfman8326 0]

Hummingbird appears to work! (I was having issues with dbus, so did not get to test bluetit; will try to resolve that another time). More details below

Here is my systemd-resolved status (prior to using hummingbird). I am on Arch Linux. I couldn't find if I'm using on-link mode or not; here is my status. I saw that Fedora has it as the default, so maybe Arch does too?
 

[me@hostname ~]$ systemctl status systemd-resolved
* systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-01-22 23:35:20 EST; 17h ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 757 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 19119)
     Memory: 5.7M
     CGroup: /system.slice/systemd-resolved.service
             `-757 /usr/lib/systemd/systemd-resolved

Warning: some journal files were not opened due to insufficient permissions.

Additionally I found this report on systemd that may or may not be related?
https://github.com/systemd/systemd/issues/5755
 

poettering commented on Apr 25, 2017

[...]"However, in contrast to classic nss-dns we have memory: when we noticed that a DNS server didn't respond or returned some failure, or for some other reason wasn't working for us, and we skip to the next, then we remember that and the next lookup is attempted with the new one. If that one fails too, then we'll skip to the next one and the next one and so on, until we reach the end of the list and start from the beginning of the list again."[...]

This almost sounds like because I had the AIrVPN dns setup all times from NetworkManager, systemd tries it, it fails (because I'm not connected to vpn), and then when I do eventually connect, it "remembers" that it failed? I could be misinterpreting this.



Potential other (minor) bug in Hummingbird; it took a little finagling to get disconnects behaving properly; it appears when I disconnect, it resets my "global" DNS settings in "resolvectl status" back to how they were prior, BUT it does not reset my "Link 2" (wired connection) settings back to how they were. This is ok for my needs, since I just reconfigured my global via resolv.conf.

I didn't record the logs of this, but if it's something you would like me to retrieve in case you need to adjust the Hummingbird code, I would be happy to get those logs for you.


Thank you again for all the help both of you! I'll come back if I encounter any more weird scenarios.

[Staff 10014]

@thewolfman8326

Hello!

Thanks for the report. It's possible that you have a Fedora 33-like setup, i.e. systemd-resolved configured to bypass resolv.conf, and network-manager also running. It seems that in this setup Hummingbird and Bluetit can properly handle DNS push (apart from the glitch you warn us about and which we'll take care the devs are informed about) but update-systemd-resolved script can't.

We would also recommend that you run Goldcrest+Bluetit when you have resolved D-Bus issues, as the client-daemon architecture provides you with a more robust and secure solution than Hummingbird does.

Kind regards