Random slow speeds upon new connections to server clusters

[staringatthesun 0]

Hello, I've used AirVPN on and off for sufficient time to report. Overall it's been very good and the connection options are the highlight.

 

Recently whenever I connect to a cluster of VPN servers - any server in Alblasserdam, NL for example - there is a 3/4 chance that the speeds are terrible - as low as 50Kbyte/s - unusable. If I disconnect and reconnect a few times to the same server or one in the same cluster, within 10 tries I get a connection that's full speed, or 3Mbyte/s and over.

 

This only happens on servers from clusters. It never happens on the locations that only have one server - those are always reliable. From the stats, none of these servers appear saturated.

 

I connect over SSH and SSL using OpenVPN client with generated configs.

 

My ISP has never throttled traffic noticeably except for OpenVPN protocol itself, which SSH and SSL circumvent. Using those protocols the speeds are well above my needs, that is, when it works.

 

The result is I am limited to a smaller part of the Air network if I want to avoid disconnecting and reconnecting 10 times each time I connect. Each time I connect to those clusters is a gamble with my time.

 

Are there settings I can try with openvpn the next time this occurs? Would forcing buffer sizes have any effect? I have no idea how they interact with SSH and SSL tunneling.

 

[Khariz 109]

There are most definitely some really wonky servers out there.  I like to use Tegmen as an example.  It's a "Canada" server, with a Seychelles IP addresss (and Seychelles latency from my location.  I get 50ms ping to canada and 300ms ping to Seychelles.  It's in Seychelles.)  Also, speed wise, you are lucky to squeak out 5-10kbps.

[OpenSourcerer 1441]

Paste us some logs to begin with. Use spoiler tags, a how-to can be found in my signature.

[zhang888 1066]

There are most definitely some really wonky servers out there.  I like to use Tegmen as an example.  It's a "Canada" server, with a Seychelles IP addresss (and Seychelles latency from my location.  I get 50ms ping to canada and 300ms ping to Seychelles.  It's in Seychelles.)  Also, speed wise, you are lucky to squeak out 5-10kbps.

 

It's not in Seychelles

For some reason, Amanah got "stuck" with a few Afrinic IPs and this is the result.

 

 

 

Recently whenever I connect to a cluster of VPN servers - any server in Alblasserdam, NL for example - there is a 3/4 chance that the speeds are terrible - as low as 50Kbyte/s - unusable. If I disconnect and reconnect a few times to the same server or one in the same cluster, within 10 tries I get a connection that's full speed, or 3Mbyte/s and over.

 

This only happens on servers from clusters. It never happens on the locations that only have one server - those are always reliable. From the stats, none of these servers appear saturated.

 

Can you provide a speedtest from 2 NL servers that have the same amount of current users and load?

Not sure why that would made such a big difference like you describe.

[Khariz 109]

Well, explain the pings then? 50ms to all other Amanah Canada servers, 300ms ping to that one. Doesn't make sense.

[zhang888 1066]

Your ISP might be using geolocation routing policy.

Terrible approach in most cases but I saw that too, especially in places

where domestic traffic is cheaper than international one.

 

Post 2 traceroutes and this will explain the issue.

 

OP:

Same applies to you.

Can you post traceroute to NL servers? One "good" and one "bad" traceroute

should probably indicate an issue on your end. Otherwise there will be hunderds

of threads with NL servers speed issue.

[Khariz 109]

Actually, my traceroutes are eerily identical for two Amanah servers. I tested both Tyl and Tegmen. I got 51ms ping to each. Functionally no difference on any single hop on either server. I don't get it. So when I traceroute, they appear to be in the same location, but when I'm hooked up to it for VPN service, it's dog butt slow, horrible pings to everything, and rock bottom speed.

 

Those don't seem to mesh.

 

http://i.imgur.com/Hdno8Gm.jpg

[staringatthesun 0]

I was going to record logs and traces this weekend but curiously the problem hasn't happened in the last couple days. It's been an ongoing problem for many months so it would be a hell of a coincidence for it to suddenly be fixed now (doubt it). When it happens again I'll post more info, but I wanted to put it out there.

 

All I can say more is the problem happened across a variety of linux distros and openvpn, ssh and stunnel versions including the most recent of each.

[staringatthesun 0]

The problem no longer seems to happen. I have no explanation, other than frequent software updates.

[staringatthesun 0]

Got the problem again today. Still a mystery.

 

Traceroutes -

178.238.AAA.AAA is a good AirVPN node, fast speed always (up to 8 megabytes/s easily)

213.152.BBB.BBB was a slow node (max couple hundred kilobytes/s)

 

stunnel/openvpn is for 213.152.BBB.BBB

 

stunnel:

2016.05.06 21:45:06 LOG5[ui]: stunnel 5.31 on XXXXXXXXXXX-linux-gnueabihf platform
2016.05.06 21:45:06 LOG5[ui]: Compiled with OpenSSL 1.0.2g  1 Mar 2016
2016.05.06 21:45:06 LOG5[ui]: Running  with OpenSSL 1.0.2h  3 May 2016
2016.05.06 21:45:06 LOG5[ui]: Update OpenSSL shared libraries or rebuild stunnel
2016.05.06 21:45:06 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI
2016.05.06 21:45:06 LOG5[ui]: Reading configuration from file XXXXXXXXXXXXXX/AirVPN_NL-Alblasserdam_Mirach_SSL-443.ssl
2016.05.06 21:45:06 LOG5[ui]: UTF-8 byte order mark not detected
2016.05.06 21:45:06 LOG5[ui]: FIPS mode disabled
2016.05.06 21:45:06 LOG6[ui]: Initializing service [openvpn]
2016.05.06 21:45:06 LOG5[ui]: Configuration successful
2016.05.06 21:45:21 LOG5[0]: Service [openvpn] accepted connection from 127.0.0.1:54369
2016.05.06 21:45:21 LOG6[0]: s_connect: connecting 213.152.162.70:443
2016.05.06 21:45:21 LOG5[0]: s_connect: connected 213.152.162.70:443
2016.05.06 21:45:21 LOG5[0]: Service [openvpn] connected remote server from 192.168.XXX.XXX:36471
2016.05.06 21:45:21 LOG6[0]: SNI: sending servername: 213.152.162.70
2016.05.06 21:45:21 LOG6[0]: CERT: Locally installed certificate matched
2016.05.06 21:45:21 LOG5[0]: Certificate accepted at depth=0: C=IT, ST=Italy, L=Perugia, O=AirVPN, OU=stunnel, CN=stunnel.airvpn.org, emailAddress=info@airvpn.org
2016.05.06 21:45:21 LOG6[0]: Client certificate not requested
2016.05.06 21:45:21 LOG6[0]: SSL connected: new session negotiated
2016.05.06 21:45:21 LOG6[0]: Negotiated TLSv1.2 ciphersuite ECDHE-RSA-AES256-GCM-SHA384 (256-bit encryption)
2016.05.06 21:46:06 LOG6[cron]: Executing cron jobs
2016.05.06 21:46:06 LOG6[cron]: Cron jobs completed in 0 seconds

openvpn:

Fri May  6 21:45:21 2016 OpenVPN 2.3.10 XXXXXXXXXX-linux-gnueabihf [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Mar 20 2016
Fri May  6 21:45:21 2016 library versions: OpenSSL 1.0.2h  3 May 2016, LZO 2.09
Fri May  6 21:45:21 2016 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Fri May  6 21:45:21 2016 WARNING: file 'user.key' is group or others accessible
Fri May  6 21:45:21 2016 WARNING: file 'ta.key' is group or others accessible
Fri May  6 21:45:21 2016 Control Channel Authentication: using 'ta.key' as a OpenVPN static key file
Fri May  6 21:45:21 2016 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri May  6 21:45:21 2016 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri May  6 21:45:21 2016 Socket Buffers: R=[87380->327680] S=[16384->327680]
Fri May  6 21:45:21 2016 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Fri May  6 21:45:21 2016 Attempting to establish TCP connection with [AF_INET]127.0.0.1:1413 [nonblock]
Fri May  6 21:45:21 2016 TCP connection established with [AF_INET]127.0.0.1:1413
Fri May  6 21:45:21 2016 TCPv4_CLIENT link local: [undef]
Fri May  6 21:45:21 2016 TCPv4_CLIENT link remote: [AF_INET]127.0.0.1:1413
Fri May  6 21:45:21 2016 TLS: Initial packet from [AF_INET]127.0.0.1:1413, sid=47e1357f b2859a28
Fri May  6 21:45:21 2016 VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org
Fri May  6 21:45:21 2016 Validating certificate key usage
Fri May  6 21:45:21 2016 ++ Certificate has key usage  00a0, expects 00a0
Fri May  6 21:45:21 2016 VERIFY KU OK
Fri May  6 21:45:21 2016 Validating certificate extended key usage
Fri May  6 21:45:21 2016 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Fri May  6 21:45:21 2016 VERIFY EKU OK
Fri May  6 21:45:21 2016 VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=server, emailAddress=info@airvpn.org
Fri May  6 21:45:22 2016 Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Fri May  6 21:45:22 2016 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri May  6 21:45:22 2016 Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Fri May  6 21:45:22 2016 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri May  6 21:45:22 2016 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Fri May  6 21:45:22 2016 [server] Peer Connection Initiated with [AF_INET]127.0.0.1:1413
Fri May  6 21:45:24 2016 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Fri May  6 21:45:25 2016 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 10.50.0.1,comp-lzo no,route-gateway 10.50.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.50.3.209 255.255.0.0'
Fri May  6 21:45:25 2016 OPTIONS IMPORT: timers and/or timeouts modified
Fri May  6 21:45:25 2016 OPTIONS IMPORT: LZO parms modified
Fri May  6 21:45:25 2016 OPTIONS IMPORT: --ifconfig/up options modified
Fri May  6 21:45:25 2016 OPTIONS IMPORT: route options modified
Fri May  6 21:45:25 2016 OPTIONS IMPORT: route-related options modified
Fri May  6 21:45:25 2016 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Fri May  6 21:45:25 2016 ROUTE_GATEWAY 192.168.XXX.YYY/255.255.255.0 IFACE=eth0 HWADDR=XXXXXXXXXXXXXXXXXXXX
Fri May  6 21:45:25 2016 TUN/TAP device tun0 opened
Fri May  6 21:45:25 2016 TUN/TAP TX queue length set to 100
Fri May  6 21:45:25 2016 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri May  6 21:45:25 2016 /usr/bin/ip link set dev tun0 up mtu 1500
Fri May  6 21:45:25 2016 /usr/bin/ip addr add dev tun0 10.50.3.209/16 broadcast 10.50.255.255
Fri May  6 21:45:25 2016 XXXXXXXXX/vpn-up-down tun0 1500 1560 10.50.3.209 255.255.0.0 init

traceroutes

# traceroute -T 213.152.BBB.BBB
traceroute to 213.152.BBB.BBB (213.152.BBB.BBB), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.218 ms  0.186 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  26.200 ms  24.658 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  20.970 ms  29.164 ms  29.301 ms
 8  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  22.030 ms  22.064 ms  22.111 ms
 9  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  24.349 ms  21.621 ms  21.584 ms
10  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  22.075 ms  25.171 ms  27.371 ms
11  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  31.437 ms * *
12  * * *
13  * * *
14  * * *
15  * * *
16  eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  120.731 ms  118.728 ms  109.279 ms
17  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  112.644 ms  112.452 ms  112.671 ms
18  . (213.152.BBB.BBB)  113.875 ms  116.287 ms  115.022 ms

# traceroute -T 178.238.AAA.AAA
traceroute to 178.238.AAA.AAA (178.238.AAA.AAA), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.220 ms  0.188 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  20.754 ms  21.336 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  21.797 ms  21.779 ms  28.681 ms
 8  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  22.562 ms  23.409 ms  23.690 ms
 9  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  25.326 ms  25.371 ms  25.538 ms
10  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  26.752 ms  26.339 ms  26.875 ms
11  * XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  48.904 ms *
12  * * *
13  * * *
14  * * *
15  * * *
16  ip-55-229-238-178.static.contabo.net (178.238.AAA.AAA)  128.803 ms  127.305 ms  126.302 ms


# traceroute -I 213.152.BBB.BBB
traceroute to 213.152.BBB.BBB (213.152.BBB.BBB), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.354 ms  0.499 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  24.375 ms  24.415 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  19.995 ms  20.057 ms  20.852 ms
 8  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  26.133 ms * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  115.351 ms  114.998 ms  115.120 ms
14  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  112.652 ms  113.420 ms  113.605 ms
15  . (213.152.BBB.BBB)  115.315 ms  115.451 ms  115.955 ms

# traceroute -I 178.238.AAA.AAA
traceroute to 178.238.AAA.AAA (178.238.AAA.AAA), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.237 ms  0.215 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  24.237 ms  24.292 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  30.359 ms  31.179 ms *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * gw03.contabo.net (62.140.24.126)  140.983 ms  140.932 ms
13  ip-55-229-238-178.static.contabo.net (178.238.AAA.AAA)  141.723 ms  141.884 ms  141.965 ms



# traceroute -U 213.152.BBB.BBB
traceroute to 213.152.BBB.BBB (213.152.BBB.BBB), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.722 ms  0.930 ms  1.185 ms
 2  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  10.811 ms  15.663 ms  16.720 ms
 3  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  17.192 ms  17.126 ms  17.385 ms
 4  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 30.408 ms  30.341 ms  30.715 ms
 5  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 28.266 ms  29.053 ms  37.029 ms
 6  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  31.656 ms  25.439 ms  24.719 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  34.365 ms  22.731 ms  20.188 ms
 8  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  29.680 ms * *
 9  et-02-00-01.core02.lon01.uk.as5580.net (78.152.53.224)  113.868 ms  114.154 ms  113.650 ms
10  ae01.core02.ams02.nl.as5580.net (78.152.53.229)  114.953 ms  117.755 ms  117.355 ms
11  * * *
12  eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  120.725 ms  122.216 ms  121.285 ms
13  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  111.470 ms  110.517 ms  109.632 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
...

# traceroute -U 178.238.AAA.AAA
traceroute to 178.238.AAA.AAA (178.238.AAA.AAA), 30 hops max, 60 byte packets
 1  XXXXXXXXX (192.168.XXX.XXX)  0.769 ms  0.953 ms  1.227 ms
 2  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  10.870 ms  13.597 ms  14.768 ms
 3  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  17.228 ms  17.167 ms  17.426 ms
 4  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 17.977 ms  26.283 ms  26.456 ms
 5  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  30.277 ms  30.213 ms  30.882 ms
 6  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 38.221 ms  34.977 ms  35.171 ms
 7  if-ae-18-0.tcore1.CT8-Chicago.as6453.net (64.86.78.21)  44.891 ms if-ae-12-0.tcore2.CT8-Chicago.as6453.net (64.86.79.89)  45.904 ms  41.082 ms
 8  if-ae-22-2.tcore1.CT8-Chicago.as6453.net (64.86.79.2)  45.262 ms  45.170 ms ae7.edge3.chicago3.level3.net (4.68.110.193)  33.180 ms
 9  ae7.edge3.chicago3.level3.net (4.68.110.193)  33.102 ms  33.191 ms ae-1-19.bar1.Munich1.Level3.net (4.69.153.245)  143.972 ms
10  * * ae-1-19.bar1.Munich1.Level3.net (4.69.153.245)  147.787 ms
11  * * *
12  * * *
13  * * *
14  * * *
...

[staringatthesun 0]

 

This is a third node, this was in NL - 213.152.CCC.CCC. It was temporarily fast but I can't know if these routes matched the fast connection for certain.

 

These don't make for great comparisons

# traceroute -T 213.152.CCC.CCC
traceroute to 213.152.CCC.CCC (213.152.CCC.CCC), 30 hops max, 60 byte packets
 1  XXXXXXXX (192.168.XXX.XXX)  0.508 ms  0.846 ms  1.173 ms
 2  XXXXXXXXXXXXXXXXXXXXXXXXXX  11.796 ms  15.724 ms *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * XXXXXXXXXXXXXXXXXXXXXXXXXX  16.963 ms  23.312 ms
 8  XXXXXXXXXXXXXXXXXXXXXXXXXX  21.575 ms  21.679 ms  22.131 ms
 9  XXXXXXXXXXXXXXXXXXXXXXXXXX  22.847 ms  23.019 ms  23.121 ms
10  eth6-7.edge1.nyc4.us.as5580.net (78.152.53.60)  35.413 ms * *
11  * * *
12  * * *
13  * * *
14  * * *
15  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  122.184 ms  108.692 ms  108.835 ms
16  . (213.152.CCC.CCC)  117.317 ms  115.327 ms  116.796 ms

# traceroute -I 213.152.CCC.CCC
traceroute to 213.152.CCC.CCC (213.152.CCC.CCC), 30 hops max, 60 byte packets
 1  XXXXXXXXXXXXXXX (192.168.XXX.XXX)  0.405 ms  0.620 ms  0.961 ms
 2  XXXXXXXXXXXXXXXXXXXXXXXXXX  16.251 ms  16.485 ms *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * XXXXXXXXXXXXXXXXXXXXXXXXXX  19.435 ms  19.550 ms
 8  * * *
 9  et-02-00-01.core02.lon01.uk.as5580.net (78.152.53.224)  111.599 ms  112.761 ms  113.061 ms
10  ae01.core02.ams02.nl.as5580.net (78.152.53.229)  114.063 ms  118.974 ms  119.263 ms
11  * * *
12  eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  129.420 ms  130.331 ms  109.888 ms
13  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  121.720 ms  106.109 ms  111.871 ms
14  . (213.152.CCC.CCC)  111.092 ms  110.307 ms  106.494 ms

# traceroute -U 213.152.CCC.CCC
traceroute to 213.152.CCC.CCC (213.152.CCC.CCC), 30 hops max, 60 byte packets
 1  XXXXXXXXXXXX (192.168.XXX.XXX)  0.447 ms  0.722 ms  1.077 ms
 2  XXXXXXXXXXXXXXXXXXXXXXXXXX  12.898 ms  16.521 ms  17.704 ms
 3  XXXXXXXXXXXXXXXXXXXXXXXXXX  25.192 ms  25.408 ms  25.582 ms
 4  XXXXXXXXXXXXXXXXXXXXXXXXXX  21.972 ms  29.087 ms  29.385 ms
 5  XXXXXXXXXXXXXXXXXXXXXXXXXX  36.410 ms  36.624 ms  36.850 ms
 6  XXXXXXXXXXXXXXXXXXXXXXXXXX  32.994 ms  26.066 ms  32.544 ms
 7  XXXXXXXXXXXXXXXXXXXXXXXXXX  32.893 ms  18.739 ms  18.906 ms
 8  * XXXXXXXXXXXXXXXXXXXXXXXXXX  32.511 ms  32.695 ms
 9  et-02-00-01.core02.lon01.uk.as5580.net (78.152.53.224)  113.502 ms  113.611 ms  220.820 ms
10  ae01.core02.ams02.nl.as5580.net (78.152.53.229)  113.442 ms  222.415 ms  222.428 ms
11  * eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  224.050 ms *
12  eth2-4.edge1.rtd1.nl.as5580.net (78.152.44.32)  222.557 ms  224.209 ms  224.043 ms
13  global-layer-57172-gw.alb01-1.nl.as5580.net (78.152.40.226)  205.545 ms  192.060 ms  191.984 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *