Incoming connections - curious behaviour(privacy?)

[thigger 0]

Not a problem, just something I'm curious about:

Incoming connections seem to always come from the same ip address (one similar to the VPN entry-ip) rather than the ip actually used to make the connection.

eg - running a server on a forwarded port, when friends from different locations try to connect I see connections all coming from the same address rather than their real addresses.

I assume this is intentional as everything works fine - is it part of the 'added security' I've seen mentioned? Does it make torrent clients misbehave if they think they're getting multiple connections from the same ip? (I know some ship by default only allowing one connection per ip).

[Staff 9972]

Hello!

Yes, it's normal. You connect to a VPN server IP address (entry-IP) and you are visible on the Internet with another IP address (the exit-IP of the same server). It is not mandatory for OpenVPN (or any VPN in general) to have its clients connected to an IP address and get out with another (i.e entry-IP and exit-IP may be the same, as it happens in almost every VPN service), anyway this is an addition, completely transparent for clients, that we have made few months ago to enhance the anonymity layer (it makes correlation attacks harder and has the benefit to avoid out-of-the-tunnel unencrypted transmissions between two clients connected to the same server which communicate to each other).

NAT and masquerading do all the job, so that your listening services (web server, torrent client...) are reachable from the outside (on the forwarded port(s)) and can respond properly to the incoming packets.

When you receive an incoming TCP connection, you should see a packet directed to your TUN IP address (assigned by the VPN DHCP - see also https://airvpn.org/index.php?option=com_content&view=article&id=74&Itemid=141).

Just a question, why do you say that you see connections coming from an address "similar to the entry-IP"? The incoming connections to your TUN interface should come from the entry-IP address. For example, on Lyra, you should see TCP IN of the type 62.x.x.85:-->10.x.x.x: (you can see your exit-IP just watching at the address reported in the bottom central box of our website while you are connected to a VPN server).

Do not hesitate to contact us for any further information.

Kind regards

[thigger 0]

Thanks for the explanation. Sorry - I should have looked closer at the ip addresses; I'd noticed they were similar but hadn't spotted they were identical!

[thigger 0]

Another query along similar lines - does the port forwarding forward udp and tcp, or just tcp? And with udp does it do the source-ip-rewriting as above?

EDIT: Just tested myself using echotool and I think I've found a problem:

tcp forwarding works fine as expected - the server sees a connection coming from the vpn entry-ip, but responses are correctly routed back to the client

udp forwarding appears to be broken in one direction - the server sees packets arriving from the vpn entry-ip, but responses never appear back at the client's end.

Could you possibly look into this?

[Staff 9972]

thigger wrote:

Does it make torrent clients misbehave if they think they're getting multiple connections from the same ip? (I know some ship by default only allowing one connection per ip).

Hello!

Sorry, this question remained unanswered in the previous message.

Torrent clients and any other listening service are not getting multiple connections from the same IP address, they see the real origin IP address. When you see packets coming from one of our VPN servers, the "original" header is encrypted together with the payload (otherwise your ISP could see the real origin with DPI). That's why you observe all packets coming from the entry-IP address of the VPN server you're connected to.

The packets are decrypted by your OpenVPN before they arrive to the listening service. You can easily verify by monitoring the traffic on your TUN/TAP interface, instead of your Ethernet or wireless interface. An excellent tool to do that is Wireshark.

Kind regards

[Staff 9972]

thigger wrote:

Another query along similar lines - does the port forwarding forward udp and tcp, or just tcp? And with udp does it do the source-ip-rewriting as above?

EDIT: Just tested myself using echotool and I think I've found a problem:

tcp forwarding works fine as expected - the server sees a connection coming from the vpn entry-ip, but responses are correctly routed back to the client

udp forwarding appears to be broken in one direction - the server sees packets arriving from the vpn entry-ip, but responses never appear back at the client's end.

Could you possibly look into this?

Hello!

Thank you for the warning. Remote forwarding is enabled both for TCP and UDP ports. You should be able to see that TCP packets for your web server have the correct header when they reach it (confirmed by the correct web server answers). We'll anyway look into this. Which server are you connected to?

Can you please elaborate about UDP packets? UDP is a connectionless protocol, and http and https are not expected to be run over UDP, so why should your web server receive UDP packets?

We're looking forward to hearing from you.

Kind regards

[thigger 0]

Thanks for looking into it. I'm not trying to get an http server to use udp (!) - I'm testing with echotool (http://bansky.net/echotool/) which is a windows tool though there are similar tools on linux (or you can knock something up in python pretty quickly!)

I've tried 4 tests, using Sirius:

1 - server on vpn'd machine, client on normal machine, udp - does not work. Packets are received on the vpn'd machine (appearing to come from the vpn entry ip) but replies don't make it back to the normal machine

2 - server on vpn'd machine, client on normal machine, tcp - works as expected using forwarded port (but connections appear to come from the vpn entry ip)

3 - server on normal machine, client on vpn'd machine, tcp - works as expected (nb normal machine is behind a NAT with the port forwarded for tcp and udp)

4 - server on normal machine, client on vpn'd machine, udp - works as expected (port forwarded as in #3)

So it definitely seems to be a problem with the forwarding setup as udp packets are making it in but not back out again.

Logs enclosed [file name=2012_jan_airvpn_udp_port_forwarding.txt size=3914]http://airvpn.org/media/kunena/attachments/legacy/files/2012_jan_airvpn_udp_port_forwarding.txt[/file]

2012_jan_airvpn_udp_port_forwarding.txt

[thigger 0]

Torrent clients and any other listening service are not getting multiple connections from the same IP address, they see the real origin IP address. When you see packets coming from one of our VPN servers, the "original" header is encrypted together with the payload (otherwise your ISP could see the real origin with DPI). That's why you observe all packets coming from the entry-IP address of the VPN server you're connected to.

That's the case with all VPN connections - but what I was wondering about is the specific feature of AirVPN where incoming packets on the tun0 interface (not the underlying eth0 interface) are masqueraded to appear to be coming from the vpn entry ip as mentioned above:

The incoming connections to your TUN interface should come from the entry-IP address. For example, on Lyra, you should see TCP IN of the type 62.x.x.85:<anyport>-->10.x.x.x:<forwarded port>

I've had a look into it, and it appeared my torrent client <em>was</em> rejecting incoming tcp connections because they had the same ip address as previous ones (and not getting any udp ones, but that's in my previous post). The solution for uTorrent was to enable 'bt.allow_same_ip' in the advanced preferences; then you can see multiple incoming connections - all appearing to have the ip of the vpn server's entry point.

[Staff 9972]

thigger wrote:

Thanks for looking into it. I'm not trying to get an http server to use udp (!) - I'm testing with echotool (http://bansky.net/echotool/) which is a windows tool though there are similar tools on linux (or you can knock something up in python pretty quickly!)

I've tried 4 tests, using Sirius:

1 - server on vpn'd machine, client on normal machine, udp - does not work. Packets are received on the vpn'd machine (appearing to come from the vpn entry ip) but replies don't make it back to the normal machine

2 - server on vpn'd machine, client on normal machine, tcp - works as expected using forwarded port (but connections appear to come from the vpn entry ip)

3 - server on normal machine, client on vpn'd machine, tcp - works as expected (nb normal machine is behind a NAT with the port forwarded for tcp and udp)

4 - server on normal machine, client on vpn'd machine, udp - works as expected (port forwarded as in #3)

So it definitely seems to be a problem with the forwarding setup as udp packets are making it in but not back out again.

Logs enclosed [file name=2012_jan_airvpn_udp_port_forwarding.txt size=3914]http://airvpn.org/media/kunena/attachments/legacy/files/2012_jan_airvpn_udp_port_forwarding.txt[/file]

Hello and thanks!

Ok, so we are going to investigate about case 1. We'll keep you posted!

Kind regards

[Staff 9972]

thigger wrote:

Torrent clients and any other listening service are not getting multiple connections from the same IP address, they see the real origin IP address. When you see packets coming from one of our VPN servers, the "original" header is encrypted together with the payload (otherwise your ISP could see the real origin with DPI). That's why you observe all packets coming from the entry-IP address of the VPN server you're connected to.

That's the case with all VPN connections - but what I was wondering about is the specific feature of AirVPN where incoming packets on the tun0 interface (not the underlying eth0 interface) are masqueraded to appear to be coming from the vpn entry ip as mentioned above:

The incoming connections to your TUN interface should come from the entry-IP address. For example, on Lyra, you should see TCP IN of the type 62.x.x.85:-->10.x.x.x:

I've had a look into it, and it appeared my torrent client was rejecting incoming tcp connections because they had the same ip address as previous ones (and not getting any udp ones, but that's in my previous post). The solution for uTorrent was to enable 'bt.allow_same_ip' in the advanced preferences; then you can see multiple incoming connections - all appearing to have the ip of the vpn server's entry point.

Hello again!

That does not appear normal at all. uTorrent should be working swiftly with bt.allow_same_ip set to "false". On the other hand, a web server could not work if the encrypted packets header were overwritten with the VPN server address, because the webserver would not know to which IP address to answer (while your web server behind AirVPN works fine even on Sirius, right?). Is it possible that your uTorrent tries to listen to your ethernet or wireless adapter, trying to bypass the tun interface on the incoming connections?

Can you please do packet capture on your TUN/TAP interface to verify whether the packets come from the correct IP address (you should see packets coming from addressess different of the VPN IP addresses).

Kind regards

[thigger 0]

That does not appear normal at all. uTorrent should be working swiftly with bt.allow_same_ip set to "false".

It does - very swiftly - but if you look closely you'll see that the incoming connections all appear to have the ip address of the entry point of the VPN server. If you have allow_same_ip set to 'false' you'll probably only have one incoming connection (identified by 'I' in the flags column).

a web server could not work if the encrypted packets header were overwritten with the VPN server address, because the webserver would not know to which IP address to answer (while your web server behind AirVPN works fine even on Sirius, right?)

It can work - so long as the VPN server keeps track of the connections and rewrites the header on the way back (we're talking about packets on the tun0 interface here, which appear as unencrypted - they're carried encrypted inside packets on the eth0 interface). This is NAT (network address translation) - the same process which allows all users of a VPN to share a single ip. As the admin in the second post stated, this use of NAT in both directions is a change made at AirVPN a few months ago. I think it's a really neat idea, but it does have the side-effect of confusing servers (and torrent clients) as it ends up that all incoming connections appear to come from the VPN rather than their real source. So I'd suggest everyone allow multiple connections from the same ip - I've certainly seen an increase in incoming connections (from 1!) since doing so.

If you still don't believe me - connect to the VPN and seed a torrent in uTorrent with port forwarding set up correctly. When you get an incoming tcp connection, check the ip (best if you have 'resolve ips' unticked on the peer tab). Then compare it with the entry ip for the VPN you're using. If you set by.allow_same_ip true, you'll find multiple incoming connections all appearing to be from the VPN entry ip. As mentioned by the admin in the 2nd post, this is intentional.

The only problem with this otherwise very fine system is that the NAT doesn't seem to work properly for UDP at the moment, so incoming datagrams can't be answered. However, it sounds like this is being looked into and should be fixed! (thanks for investigating btw - let me know if you need any more logs; sorry the tool I'm using is windows-based!)

[Staff 9972]

thigger wrote:

The only problem with this otherwise very fine system is that the NAT doesn't seem to work properly for UDP at the moment, so incoming datagrams can't be answered. However, it sounds like this is being looked into and should be fixed! (thanks for investigating btw - let me know if you need any more logs; sorry the tool I'm using is windows-based!)

Hello!

Problem detected and fixed on all servers.

Now you'll see incoming packets from the exit-IP, not the entry-IP. In this way the UDP reply will be routed inside the tunnel (if the reply was routed to the entry-IP, it would go unencrypted outside the tunnel to the entry-IP, and then dropped by the VPN servers - this was the cause of the problem). We apologize for the inconvenience.

Of course, you will still be able to see real IP sources and destinations by capturing packets on the tun interface.

Thank you for your cooperation and let us know if everything's fine now!

Kind regards

[thigger 0]

Excellent, thanks - seems to be working fine now!

I'm curious that you say the masquerading doesn't happen for you on incoming connections - I thought (in post #2) you'd said this was a deliberate plan to avoid correlation attacks? As it is, here (on tun0) all incoming tcp connections and unsolicited UDP packets have their ip masqueraded to appear to be coming from the exit-ip of the VPN (was previously the entry-ip). So there's no way for a web server/torrent client or anything else to know the originating ip; and using uTorrent at least it's necessary to allow multiple connections from the same ip.

Screenshot enclosed of what I mean - working fine (there's 4 TCP connections and 1 UDP one there) but all incoming connections have the exit-ip of the VPN.

On eth0 there aren't any incoming connections at all, just the udp (or tcp) stream established as an outgoing connection from this computer to the entry-ip of the VPN.

So certainly for me it's best to leave bt.allow_same_ip as 'true' - I'd be interested to know if anyone else has similar results (to test: look for 'I' in the 'flags' column, and you should find they all have the ip of the exit-point of the VPN).

Thanks for fixing this so quickly!

[Staff 9972]

thigger wrote:

Excellent, thanks - seems to be working fine now!

I'm curious that you say the masquerading doesn't happen for you on incoming connections - I thought (in post #2) you'd said this was a deliberate plan to avoid correlation attacks? As it is, here (on tun0) all incoming tcp connections and unsolicited UDP packets have their ip masqueraded to appear to be coming from the exit-ip of the VPN (was previously the entry-ip). So there's no way for a web server/torrent client or anything else to know the originating ip; and using uTorrent at least it's necessary to allow multiple connections from the same ip.

Hello!

Yes, you're 100% right: forget the previous incorrect statement. And yes, a nice side effect of the setup is to make correlation attacks much more difficult.

We'll also add in the FAQ your suggestion for uTorrent which can improve performance. Technically, we don't do masquerading (in the meaning of MASQUERADE in iptables), but DNAT and SNAT (which is a form of masquerading according to some older definitions; in some literature, dynamic NAT is also called masquerading).

Thanks for fixing this so quickly!

Thank you for your cooperation!

Kind regards

[thigger 0]

Just one more question...

A friend and I were trying to work out the nature of the correlation attack you're protecting against with this behaviour. I appreciate you may not want to give away details if it involves information about AirVPN's internal setup.

We figured that on the leg from the client to the VPN exit, the header's unencrypted but not rewritten yet. Internally (on its way to the entry ip) it may be unencrypted but presumably that leg is on a private internal network rather than out in the open. On its arrival at the entry-ip (or perhaps before) it gets encrypted by OpenVPN so it should only be susceptible to timing-based correlation attacks - even someone able to monitor the packets shouldn't be able to read the source address whether it's been rewritten or not. At the other end of the VPN tunnel (ie someone connecting to AirVPN as a client) it's unencrypted again but is now on a local machine.

My best guess was to protect against someone running a Tor entry point over your VPN and with the ability to monitor various Tor exit nodes - with the rewriting/SNAT in place they can't identify the sender.

Alternatively, is the use of rewriting/SNAT just a side effect that's necessary because of the way the entry-ip and exit-ip computers communicate?

Sorry to be so curious but this kind of thing always interests me and it's irritating to have a puzzle I can't work out!

[Staff 9972]

thigger wrote:

Just one more question...

A friend and I were trying to work out the nature of the correlation attack you're protecting against with this behaviour. I appreciate you may not want to give away details if it involves information about AirVPN's internal setup.

Hello!

Don't worry, security through obscurity is a very dangerous path, so we try to avoid it.

We figured that on the leg from the client to the VPN exit, the header's unencrypted but not rewritten yet.

A little bit unclear... if you refer to a packet coming "from the Internet" with destination the exit-IP of a VPN server, it is not (necessarily) encrypted, it depends on how and by whom that packet was sent. If a client of ours sends a packet to the exit-IP of a VPN server (for example a client with a service which listens and replies), it is encrypted on the client side together with the payload before it gets out of the client machine. The unencrypted new header will have, as destination, the entry-IP of that VPN server.

Internally (on its way to the entry ip) it may be unencrypted but presumably that leg is on a private internal network rather than out in the open. On its arrival at the entry-ip (or perhaps before) it gets encrypted by OpenVPN so it should only be susceptible to timing-based correlation attacks - even someone able to monitor the packets shouldn't be able to read the source address whether it's been rewritten or not. At the other end of the VPN tunnel (ie someone connecting to AirVPN as a client) it's unencrypted again but is now on a local machine.

Yes, that's right.

My best guess was to protect against someone running a Tor entry point over your VPN and with the ability to monitor various Tor exit nodes - with the rewriting/SNAT in place they can't identify the sender.

Again, correct! Just in case, any second guess? Congratulations!

Kind regards

[thigger 0]

My best guess was to protect against someone running a Tor entry point over your VPN and with the ability to monitor various Tor exit nodes - with the rewriting/SNAT in place they can't identify the sender.

 

Again, correct! Just in case, any second guess? Congratulations!

We couldn't think of an obvious second attack - the rewritten header is encrypted until it reaches the AirVPN user so shouldn't be vulnerable en-route; the only places to see the header rewritten but unencrypted are at AirVPN (but you guys have access to the un-rewritten header too) and on the AirVPN user's machine - so we figured the only person losing information by the rewriting is the AirVPN user. I think that's why we found it confusing - we'd assumed any changes were to protect the anonymity of AirVPN users rather than the anonymity of TOR or other users connecting to a service run by an AirVPN user!

BTW: I am amazed at how responsive your setup is - I spent the day connected over AirVPN yesterday without noticing (openvpn helpfully autoconnected when I turned the machine on); it appeared to be just as quick as my raw connection!

[Staff 9972]

thigger wrote:

My best guess was to protect against someone running a Tor entry point over your VPN and with the ability to monitor various Tor exit nodes - with the rewriting/SNAT in place they can't identify the sender.

 

Again, correct! Just in case, any second guess? Congratulations!

We couldn't think of an obvious second attack - the rewritten header is encrypted until it reaches the AirVPN user so shouldn't be vulnerable en-route; the only places to see the header rewritten but unencrypted are at AirVPN (but you guys have access to the un-rewritten header too) and on the AirVPN user's machine - so we figured the only person losing information by the rewriting is the AirVPN user. I think that's why we found it confusing - we'd assumed any changes were to protect the anonymity of AirVPN users rather than the anonymity of TOR or other users connecting to a service run by an AirVPN user!

Hello!

That's also important, since "other users connecting to a service run by an AirVPN user" might be AirVPN users too.

Furthermore, some Air users might be malicious users.

Anyway, there are further considerations. The packets with final destination the entry-IP server address a client is connected to are sent outside the encrypted tunnel. This is a correct behaviour; however, when the entry and the exit-IP are the same, in the event in which two or more users are connected to the same VPN server and try to exchange data between their services (maybe even unaware of that), one of them or both would send out unencrypted packets.

A not uncommon situation is a p2p environment, in which the VPN server IP address and the port forwarded by the client are inserted in the DHT table and/or in a tracker peerlist. The torrent client of some (other or the same) VPN user sharing the file identified by the same torrent or magnet link would see that IP address:port as a good IP address:port to try a connection to, and would perform at least one connection attempt which would be routed outside the tunnel (if the user's firewall is not properly configured to prevent that).

Our servers (and hopefully any server of any VPN service which offers remote port forwarding) always dropped those packets for security, otherwise a copyright troll could just subscribe to the service and start seeding something apparently tasty to disclose the real IP address of the VPN users who did not set properly their firewall.

However, this is not a completely satisfactory solution, because a malicious entity which wiretaps locally the connections of a user could realize that he/she is using a certain protocol. Not a relevant information, but you never know when you deal with repressive regimes. Furthermore, some rotten ISPs perform bandwidth caps against p2p: in this case a single packet outside the tunnel might trigger the cap for seconds or even minutes.

With separate entry and exit-IP addresses, we solve simultaneously both this issue and the issue you noticed, without requiring our users any additional firewall rule and with no need to renounce to remote port forwarding.

BTW: I am amazed at how responsive your setup is - I spent the day connected over AirVPN yesterday without noticing (openvpn helpfully autoconnected when I turned the machine on); it appeared to be just as quick as my raw connection!

Thank you from the techies!

Kind regards

[perhentian 2]

Great job Support!! Really super!

[thigger 0]

admin wrote:

The packets with final destination the entry-IP server address a client is connected to are sent outside the encrypted tunnel. This is a correct behaviour; however, when the entry and the exit-IP are the same, in the event in which two or more users are connected to the same VPN server and try to exchange data between their services (maybe even unaware of that), one of them or both would send out unencrypted packets.

Yeah - we agreed it's an excellent idea to have separate entry and exit IPs; we were just confused by the SNAT/ip rewriting.

However with your clue I think I've found a better reason as to why the rewriting is a good idea. Without it, if an attacker was monitoring an AirVPN user's connection they could match the connection to a forwarded port by sending a packet to the VPN exit with the source ip spoofed to be the VPN's entry ip. The user's machine would send a reply outside the VPN tunnel, which would be seen by the attacker and confirm the user was using that port. Rewriting/SNATing means that the reply packet goes down the tunnel, preventing the attack.

Is that the main reason? Thanks for the fascinating insight into how AirVPN works.

[Staff 9972]

thigger wrote:

admin wrote:

The packets with final destination the entry-IP server address a client is connected to are sent outside the encrypted tunnel. This is a correct behaviour; however, when the entry and the exit-IP are the same, in the event in which two or more users are connected to the same VPN server and try to exchange data between their services (maybe even unaware of that), one of them or both would send out unencrypted packets.

Yeah - we agreed it's an excellent idea to have separate entry and exit IPs; we were just confused by the SNAT/ip rewriting.

However with your clue I think I've found a better reason as to why the rewriting is a good idea. Without it, if an attacker was monitoring an AirVPN user's connection they could match the connection to a forwarded port by sending a packet to the VPN exit with the source ip spoofed to be the VPN's entry ip. The user's machine would send a reply outside the VPN tunnel, which would be seen by the attacker and confirm the user was using that port. Rewriting/SNATing means that the reply packet goes down the tunnel, preventing the attack.

Is that the main reason? Thanks for the fascinating insight into how AirVPN works.

Hello!

Yes, exactly. It is a dangerous attack. One of us has also verified in the last days that this attack is possible with some VPN services. We have seen that it was also discussed in the Wilders Security Forums a couple of years ago (unfortunately no real solution was suggested at the time, except the unsatisfactory solution to prevent users to forward ports) so hopefully VPN providers are getting aware of that.

It appears to be completely prevented by our current setup. The thread is becoming more and more interesting; if you (or anybody reading) have any idea on further attacks that you think might be successful with our current setup, please do not hesitate to share. We always do our best to harden security and we keep ourselves constantly updated from security forums and bulletins, but independent peer reviews and suggestions are anyway vital. Let's take it as a challenge where nobody can lose.

Kind regards

[Staff 9972]

perhentian wrote:

Great job Support!! Really super!

Thank you very much!

Kind regards