High ping to all dutch servers

[mariusffm 0]

Hello,

i recently noticed that I suddenly have a very high ping and slow speed to all servers in the Netherlands.

Here is a MTR to the IP 213.152.161.69:
 

Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                          0.0%    40    0.7   0.7   0.5   1.3   0.1
 2. pxxxxxxxx.dip0.t-ipconnect.de     0.0%    39    4.7   3.1   1.9  22.4   3.4
 3. f-ed11-i.F.DE.NET.DTAG.DE         0.0%    39    5.4 247.3   5.3 2536. 599.6
 4. 80.156.162.158                    0.0%    39    5.0   5.3   4.8   6.0   0.2
 5. ae5-2074.ams10.core-backbone.com  0.0%    39   10.8  10.9  10.6  11.5   0.2
 6. core-backbone.global-layer.com    0.0%    39  144.8 136.0 106.2 151.8  11.4
 7. 37.123.210.79                     0.0%    39  131.9 136.4 112.1 165.3  12.4
 8. 69.161.152.213.in-addr.arpa       0.0%    39  126.5 133.1 112.4 150.4  10.2

For me it looks like the problem is between Core-Backbone and Global-Layer: I'm able to ping an IP of core-backbone without any problems.

PING ams.lg.core-backbone.com (31.7.177.131) 56(84) bytes of data.
64 bytes from ams.lg.core-backbone.com (31.7.177.131): icmp_seq=1 ttl=59 time=11.8 ms
64 bytes from ams.lg.core-backbone.com (31.7.177.131): icmp_seq=2 ttl=59 time=10.9 ms
64 bytes from ams.lg.core-backbone.com (31.7.177.131): icmp_seq=3 ttl=59 time=11.0 ms
64 bytes from ams.lg.core-backbone.com (31.7.177.131): icmp_seq=4 ttl=59 time=11.1 ms
64 bytes from ams.lg.core-backbone.com (31.7.177.131): icmp_seq=5 ttl=59 time=11.0 ms

Can someone of the mods have a look into that? Edited ... by mariusffm

[OpenSourcerer 1416]

1 hour ago, mariusffm said:

Can someone of the mods have a look into that? 

Okay, someone looks into it. What is that someone supposed to do then?
Did a few pings on hop 6, domain unknown. Pinged hop 7 instead and got 17 ms on average from my end. For now I'll say "can't reproduce".

[mariusffm 0]

Unfortunately the problems started again.  The ping is high and unstable, download and upload are slow.

 Host                                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1                                                              0.0%    45    0.8   0.8   0.6   1.1   0.1
 2. 62.155.246.254                                                           0.0%    45    2.3   2.7   1.7  21.5   2.9
 3. 217.5.118.26                                                             0.0%    45    5.6 457.7   5.5 3651. 927.1
 4. 80.156.162.158                                                           0.0%    45    5.1   5.2   4.8   5.8   0.2
 5. 81.95.2.138                                                              0.0%    44   10.7  10.7  10.3  11.5   0.2
 6. 80.255.15.206                                                            0.0%    44  106.3  90.0  74.1 106.3   7.7
 7. 37.123.210.79                                                            0.0%    44  104.1  96.2  83.3 114.0   7.0
 8. 213.152.161.69                                                           0.0%    44   91.2  88.4  73.9 105.8   7.2

It looks like the peering capacity is overloaded. The situation is changing all the time. For a few hours the connection looks good but worsens suddenly, specially during the prime time.
 

[OpenSourcerer 1416]

I don't see how anyone here can help it. Doesn't look like anything AirVPN can influence.

[mariusffm 0]

Since I'm not a customer of Global Layer I cannot contact them regarding this problem, only AirVPN can. Normally I would sent them some MTRs and asking them if they can check their peering connectivity / capacity. It could also be that the
connection is simply overbooked.

[Staff 9921]

@mariusffm

Hello!

Which is exactly the prime time you mention, when you notice higher round trip times? This is what we currently see from various NL servers (cut to the first entries) to and from a bunch of our servers (even non-VPN servers) in other datacenters served by different transit providers. It's very good for a Saturday night in Europe (ignore the duplicates like "Dublin" and "Ireland", it's just how our monitoring system organizes the outcome). Consider that such tests are performed every other minute so we can have a fine grained report and if you point us to the prime time we can see whether something is wrong (so far we have not detected peculiar problems).

Kind regards
 

	Alblasserdam 0 ms Noord-Holland 1 ms Amsterdam 1 ms Haarlem 1 ms France - Roubaix 6 ms Ile-de-France 6 ms Berlin 7 ms France - Gravelines 7 ms Germany 7 ms Frankfurt 8 ms London 10 ms Munich 14 ms Bern 15 ms Manchester 15 ms Zurich 15 ms Dublin 17 ms Ireland 17 ms Riga 18 ms Prague 19 ms Vienna 22 ms Stockholm 22 ms Belgrade 24 ms Uppsala 27 ms Madrid 27 ms Barcelona 27 ms Oslo 28 ms Siauliai 29 ms Bucharest 30 ms Arezzo 31 ms

[mariusffm 0]

@Staff I will setup smokeping for more insight. Meanwhile I noticed that the routing is asymmetric.

A traceroute while connected to a NL VPN to one of my provider IPs shows this:
 

Tracing route to p3ee26501.dip0.t-ipconnect.de [62.226.101.1]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  192.168.1.254
  2    98 ms    98 ms    97 ms  10.128.0.1
  3    85 ms    78 ms    81 ms  hosted-by.global-layer.com [134.19.179.25]
  4    70 ms    77 ms    69 ms  37.123.210.78
  5    90 ms    90 ms    89 ms  adm-b6-link.ip.twelve99.net [62.115.162.162]
  6    89 ms    91 ms    96 ms  adm-bb3-link.ip.twelve99.net [62.115.116.124]
  7    93 ms    93 ms    90 ms  hbg-bb3-link.ip.twelve99.net [213.155.136.161]
  8    92 ms    94 ms    98 ms  bei-b1-link.ip.twelve99.net [62.115.139.9]
  9   114 ms   118 ms   101 ms  bei-b2-link.ip.twelve99.net [62.115.134.195]
 10   181 ms   183 ms   168 ms  80.156.161.137
 11   175 ms   179 ms   161 ms  p5b17f011.dip0.t-ipconnect.de [91.23.240.17]
 12   169 ms   174 ms   181 ms  p3ee26501.dip0.t-ipconnect.de [62.226.101.1]

Trace complete.

I verified this with the Looking Glass of Telia, also see: https://lg.twelve99.net/?type=traceroute&router=adm-b6&address=62.226.101.1
 

Router: adm-b6 / Amsterdam (Iron Mountain, Haarlem)
Command: traceroute ipv4 62.226.101.1 timeout 1 source Loopback0

Tracing the route to 62.226.101.1

 1  adm-bb3-link.ip.twelve99.net (62.115.116.124) 16 msec  15 msec  15 msec 
 2  hbg-bb3-link.ip.twelve99.net (213.155.136.161) 14 msec 
    hbg-bb3-link.ip.twelve99.net (62.115.112.13) 14 msec  15 msec 
 3  bei-b1-link.ip.twelve99.net (62.115.139.9) 14 msec  14 msec  14 msec 
 4  bei-b2-link.ip.twelve99.net (62.115.134.195) 15 msec  14 msec  15 msec 
 5  80.156.161.137 86 msec  66 msec  95 msec 
 6  p5b17f011.dip0.t-ipconnect.de (91.23.240.17) 96 msec  68 msec  64 msec 
 7  p3ee26501.dip0.t-ipconnect.de (62.226.101.1) !A  !A  !A

It looks like the peering between DTAG AS3320 and AS1299 Telia is overloaded, which unfortunately is out of control of Global-Layer i guess. Maybe its possible to do a request for a bilateral peering.
 

[Seebarschtian 5]

On 1/8/2022 at 9:54 PM, mariusffm said:

Maybe its possible to do a request for a bilateral peering.

The problem in this setting might be up to 90% the mindset of Deutsche Telekom. They want something called "Double Paid Traffic" which means, that the customers should pay and the datacenters/transit providers/peering partners for the same traffic.
Thing here is, that the costs for connectivity should already be earned by the earnings made from the customers contract (DSL/Fiber/LTE/5G/etc).

The German Hosting provider Hetzner had a nice article about this some time ago and were fighting against that, but even they fall for the extra costs to get their customers a good connection to the Telekom. Sadly they took their article down.

TL;DR:
You should change your ISP to another one which is not artificially throttling the possible connection by not having enough backbone capacity.
Regarding to your needs and infrastructure you should take a look at Telefonica/o2. They have a good peering capacity and enable a wide range of connections (DSL/Coax/Fiber/LTE/5G).

[OpenSourcerer 1416]

8 hours ago, Seebarschtian said:

L;DR:
You should change your ISP to another one which is not artificially throttling the possible connection by not having enough backbone capacity.
Regarding to your needs and infrastructure you should take a look at Telefonica/o2. They have a good peering capacity and enable a wide range of connections (DSL/Coax/Fiber/LTE/5G).

They might have a good capacity, maybe even good peering, but they deprioritize "unknown" traffic such as encrypted OpenVPN themselves. Happened to myself with Vodafone, and I've got family on o2. You do not want to recommend that as an "improvement". The peering may improve, sure, but you get more than one more problems with VPN connectivity.

Also, this topic seems to date back to at least 2018, and even back then there was more than one opinion on it. [German link]