forwarded ports sshd not receiving connections from a forwarded port on linux mint

[King0777 0]

Hi,

I have a problem that may come from eddie or my system.
I recently switched to linux from windows, so I’m not well known in how things are working.

I’ve configured the OpenSSH server on my laptop.
I’m not using the port 22 but another port X.
It’s working when ssh-ing on it’s local IP from another device on my LAN (and also from itself on localhost).

I have eddie installed on the laptop, configured and working on the device "Laptop".

❯ sudo ip addr show dev Eddie 
10: Eddie: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1320 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet [IPv4]/32 scope global Eddie
       valid_lft forever preferred_lft forever
    inet6 [IPv6]/128 scope global 
       valid_lft forever preferred_lft forever

I have a forwarded port set in the client area from port Y :2 to port X, set to the correct device "Laptop".
I’ve set a [ABC].airdns.org DDNS on that forwarded port.
That forwarding was working when I was on windows to connect to a ssh server listening on the same port X.

I’ve added UFW rules to allow connections from Anywhere to port X for IPv4 and IPv6.

❯ sudo ufw status verbose  
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
[port X]/tcp                  ALLOW IN    Anywhere                   (log)
[port X]/tcp (v6)             ALLOW IN    Anywhere (v6)              (log)

When ssh-ing to the [ABC].airdns.org on port Y from another device not using a vpn, UFW is receiving the connection on port X and allowing it(I’ve anonymized the IP and ports in the following logs.)

2025-02-14T17:20:12.329780+01:00 laptop kernel: [UFW ALLOW] IN=Eddie OUT= MAC= SRC=[device IPv6] DST=[local Eddie IPv6] LEN=80 TC=40 HOPLIMIT=50 FLOWLBL=855788 PROTO=TCP SPT=42266 DPT=[port X] WINDOW=65535 RES=0x00 SYN URGP=0
2025-02-14T17:20:43.790702+01:00 laptop kernel: [UFW ALLOW] IN=Eddie OUT= MAC= SRC=[device IPv4] DST=[local Eddie IPv4] LEN=60 TOS=0x08 PREC=0x20 TTL=55 ID=43836 DF PROTO=TCP SPT=43944 DPT=[port X] WINDOW=65535 RES=0x00 SYN URGP=0 

sshd is not getting the connections as the output stays listening.

❯ sudo /usr/sbin/sshd -d
debug1: sshd version OpenSSH_9.6, OpenSSL 3.0.13 30 Jan 2024
debug1: private host key #0: [CENSORED]
debug1: private host key #1: [CENSORED]
debug1: private host key #2: [CENSORED]
debug1: Set /proc/self/oom_score_adj from 200 to -1000
debug1: Bind to port [PORT X] on 0.0.0.0.
Server listening on 0.0.0.0 port [PORT X].
debug1: Bind to port [PORT X] on ::.
Server listening on :: port [PORT X].

The ssh client is trying the server IPv4, then IPv6, before timing out.

❯ ssh lapa -v
OpenSSH_9.6p1 Ubuntu-3ubuntu13.5, OpenSSL 3.0.13 30 Jan 2024
debug1: Reading configuration data [HOME]/.ssh/config
debug1: [HOME]/.ssh/config line 6: Applying options for lapa
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to [ABC].airdns.org [IPv4] port [port Y].
debug1: connect to address [IPv4] port [port Y]: Connection timed out
debug1: Connecting to [ABC].airdns.org [IPv6] port [port Y].
debug1: connect to address [IPv6] port [port Y]: Connection timed out
ssh: connect to host 41r.airdns.org port 55107: Connection timed out

ssh-ing from my laptop to [ABC].airdns.org on port Y does not work either, but it’s not logged by the firewall.
Either it’s not going through the firewall because it’s routing to the local Eddie interface, or it’s blocked by the VPN config.


When trying the "Test open" functionality in the client area for that forwarded port, both IPv4 and IPv6 are shown as opened.
The sshd receives the test connections as can be seen by the output of the debug mode (only the IPv4 is shown here as the sshd quit after the first connection in debug mode).

❯ sudo /usr/sbin/sshd -d
debug1: sshd version OpenSSH_9.6, OpenSSL 3.0.13 30 Jan 2024
debug1: private host key #0: [CENSORED]
debug1: private host key #1: [CENSORED]
debug1: private host key #2: [CENSORED]
debug1: Set /proc/self/oom_score_adj from 200 to -1000
debug1: Bind to port [PORT X] on 0.0.0.0.
Server listening on 0.0.0.0 port [PORT X].
debug1: Bind to port [PORT X] on ::.
Server listening on :: port [PORT X].
debug1: Server will not fork when running in debugging mode.
Connection from 142.93.172.65 port 44700 on [IPv4] port [PORT X] rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5
kex_exchange_identification: Connection closed by remote host
Connection closed by 142.93.172.65 port 44700

It generates the following UFW logs:

2025-02-14T17:31:46.608668+01:00 laptop kernel: [UFW ALLOW] IN=Eddie OUT= MAC= SRC=142.93.172.65 DST=[local Eddie IPv4] LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=33260 DF PROTO=TCP SPT=44700 DPT=[port X] WINDOW=64240 RES=0x00 SYN URGP=0 
2025-02-14T17:31:46.616753+01:00 laptop kernel: [UFW ALLOW] IN=Eddie OUT= MAC= SRC=2a03:b0c0:0003:00d0:0000:0000:0d02:6001 DST=[local Eddie IPv6] LEN=80 TC=0 HOPLIMIT=56 FLOWLBL=721782 PROTO=TCP SPT=35088 DPT=[port X] WINDOW=64800 RES=0x00 SYN URGP=0 

I’m out of ideas where to look next.
If someone knows where the problem could lay, I would greatly appreciate.

[OpenSourcerer 1450]

If you disable UFW, does it still time out? If yes, it's not the firewall rules.
Does the DDNS name resolve to something, and if yes, is the IP address correct?

[King0777 0]

1 hour ago, OpenSourcerer said:

If you disable UFW, does it still time out? If yes, it's not the firewall rules.
Does the DDNS name resolve to something, and if yes, is the IP address correct?

Hi,
Yes it still time out with the firewall disabled.
Connecting from my local network to my LAN ip is working and the firewall logs allowing the connection.

Connecting from another device to the DDNS name and forwarded port is also allowed and logged by the firewall incoming from the Eddie interface, to the correct destination port (port X from my initial post).
But sshd does not receive that connection.
So it should not be a problem with the config of the DDNS and forwarded port.

Something is blocking the connection from the Eddie interface locally between the UFW firewall and sshd.
Either another firewall on my system (I’ve just moved to linux), or a misconfig with Eddie, or something to set in the sshd config (it’s listening on 0.0.0.0 and :: so I think not?).

I’ve just tried skipping AirVPN and doing a forwarded port on my router.
I can connect to the forwarded port from my LAN but it’s not working when my device trying to connect is outside my LAN.
So it’s not a problem with AirVPN but my system 😅
Now I have other trails to follow.

[King0777 0]

25 minutes ago, King0777 said:

So it’s not a problem with AirVPN but my system 😅
Now I have other trails to follow.

So, I think I have the same problem that
The topic was created nearly 10 years ago but if it was solved, the solution was not posted.
I’ve turned off the Network lock (unchecked "Ensure…", Mode: None)
A very similar eddie config was working on Windows but not on linux.

I’ve also found this but as the problem was solved, it should not be the same one?


To do some tests, I’ve added a port forwarding on my router to ssh in outside of the vpn.

So I have:
- Airvpn: forward IP [ABC].airdns.org [port Y1] to laptop [port X]
- LAN: router forward outer IPv4, port [Y2] to laptop [port X]
- phone with 2 ssh config
    - lapAirvpn to connect to the laptop through the Ainvpn DDNS and forwarded port Y1
    - lapDirect to connect to the laptop through my router external IP and forwarded port Y2
port Y1 ≠ port Y2 just to be sure.

When eddie is not launched on my laptop:
- I can connect to lapDirect with my phone from inside my LAN (using my local wifi) or from outside (using my cellular data).

When eddie is launched and connected on my laptop:
- I can connect to lapDirect only from **inside** my LAN. From outside, the connection timeouts.
- I can connect to lapAirvpn only from **outside** my LAN. From inside, the connection timeouts.

With some more researches, I’m pretty sure it has something to do with configuring the firewall to transfer/route packets between the virtual Eddie interface and the physical ones from my laptop (wi-fi and rj45), but I’ve not found what rules I need to set where to solve my problem.
This gist https://gist.github.com/kimus/9315140 seems promising, but I’m not sure what I need to do or not.
Well, if anyone already knows the rules I can use for my case, I would gladly accept them.
Else I will first start by using rules to completely deactivate the firewall if it fixes the problem, not loose more time to find a more "correct" solution.
Come on, I must not be the only one to have that problem, where is the official documentation from AirVPN on how to do it???

[Staff 10079]

2 hours ago, King0777 said:

Come on, I must not be the only one to have that problem, where is the official documentation from AirVPN on how to do it???

Hello!

It's here:
https://airvpn.org/faq/port_forwarding/

We can already reach your sshd (and another service too) through the proper ports of the VPN servers you're connected to, so your setup seems already correct and working.

Side note: you have also defined a *.airdns.org name to reach your sshd. Note that when you change VPN server the record will be updated immediately by our authoritative DNS server, but the TTL is 1 hour, so when you change server you may need to wait for the propagation if you query a public DNS.

Kind regards
 

[King0777 0]

1 hour ago, Staff said:

Hello!

It's here:
https://airvpn.org/faq/port_forwarding/

We can already reach your sshd (and another service too) through the proper ports of the VPN servers you're connected to, so your setup seems already correct and working.

Side note: you have also defined a *.airdns.org name to reach your sshd. Note that when you change VPN server the record will be updated immediately by our authoritative DNS server, but the TTL is 1 hour, so when you change server you may need to wait for the propagation if you query a public DNS.

Kind regards
 

Hi,

My problem is not with the configuration of the forwarded port, or the DDNS.

My problems are
- that I can’t connect in through the VPN forwarded port from a device on the same LAN.
- and I can’t connect in outside of the VPN (through a forwarded port set on my router) from a device outside of my LAN.
(Re-read my previous messages if you don’t understand that shorter explanation.)

I’ve captured the packets exchange with wireshark to be sure of the problem.
When trying to connect from another device on the same LAN through the VPN forwarded port, the SYN packet opening the TCP connection is received through the Eddie interface, as expected.
But the SYN,ACK response is sent back through my eth interface.
(I’ve not done the capture for a direct connection from outside of my LAN, but I’m pretty sure it’s the reverse situation. The SYN is received on eth, but the SYN,ACK is sent through the Eddie interface.)

That was working on Windows, it’s not working on linux.
Now there lies my problem, I don’t know how to configure my system so responses to packets received from an interface are always sent back from the same interface.

When I test my forwarded port with the "Test open" in the client area, the TCP SYN,ACK response is correctly sent back through the Eddie interface because the test connection is coming from IPv4 and v6 outside of my LAN.

Kind regards

 

[Staff 10079]

15 hours ago, King0777 said:

My problems are
- that I can’t connect in through the VPN forwarded port from a device on the same LAN.

Hello!

This is expected. More details on why it can't work later on this message.
 

15 hours ago, King0777 said:

- and I can’t connect in outside of the VPN (through a forwarded port set on my router) from a device outside of my LAN.

Expected too, due to the routing table and also the Network Lock if enabled.

If you need traffic leaks so that a listening program is reachable both on the "real" public IP address and on the VPN server exit-IP public address, you should disable Network Lock (or re-configure it accordingly), have a program listen to all interfaces, define (only when necessary) source based routing, and configure WireGuard NOT to route all traffic inside the tunnel (**).
 

15 hours ago, King0777 said:

When trying to connect from another device on the same LAN through the VPN forwarded port, the SYN packet opening the TCP connection is received through the Eddie interface, as expected.

This means that you tried to reach the listening program on the VPN interface IP address from inside the same LAN, or to the public VPN server IP address (or through the DDNS, same thing) - this is an error because then the physical reply is necessarily routed into the VPN tunnel (check your system's routing table (*) and you will immediately see why; or that WireGuard is tunneling the whole IP space (default behavior with Eddie Desktop edition). As you noted:
 

Quote

But the SYN,ACK response is sent back through my eth interface.

You must contact the service directly on the LAN and you have to make sure that WireGuard does not send everything inside the VPN tunnel (**), to avoid that the reply is routed in the VPN tunnel and becomes un-routable on the VPN server. The interpretation you give here is not totally correct.

It's true that the physical reply is routed finally through eth (just like anything else, because a virtual network sooner or later must rely on something physically existing), but the packets transiting there have been already encrypted and wrapped.

The final destination address is the original sender address through the VPN routing. The reply (the underlying payload of the traffic) then gets lost as expected because it must go to the same public IP address of your router, i.e. the same public IP address the VPN server sees your client connecting from, which is also the same public IP address the request to your listening service comes from, in the eyes of the VPN server.
 

15 hours ago, King0777 said:

That was working on Windows, it’s not working on linux.

This is very strange. Maybe you had a different setup in Windows? Anyway the correct behavior is the one you just described on Linux.

(*) Please note that if you have enabled multiple routing tables then source based routing may become strictly necessary to ensure that responses to packets received from an interface are always sent back from the same interface.  From your description this doesn't seem the case, but if it is, please keep it into consideration. You will also need that devices in the same LAN can route outgoing traffic to different public IP addresses to contact a listening service (through the VPN) of a machine in the same LAN. It doesn't make sense anyway, just pass through the LAN.

(**) The option to do so is currently not implemented in Eddie Desktop edition, but only in Eddie Android edition. It is planned anyway on an imminent AirVPN Suite for Linux release. Please see here for a thorough explanation and a possible solution:
https://airvpn.org/forums/topic/55801-wireguard-access-local-network/?do=findComment&comment=217458

Kind regards
  Edited ... by Staff
Important information related to WireGuard and added practical solutions

[NaDre 159]

17 hours ago, King0777 said:

...
- and I can’t connect in outside of the VPN (through a forwarded port set on my router) from a device outside of my LAN.
...
 

If I understand you, then this is what I do on remote Linux servers, when I want to run a VPN client on them:

https://github.com/tool-maker/VPN_just_for_torrents/wiki/Maintaining-SSH-Access-Using-a-VPN-on-a-Remote-Linux-Server

It allows me to connect to SSH through the real network interface, rather than having to go through a forwarded port on the VPN server.