ANSWERED Terrible packet loss in Canada

[Baraka 33]

I've been with AirVPN for a long, long time. I wrote the original Tomato router guide. I've never had problems like this before. The packet loss is very persistent. Kornephoros is the only node in eastern Canada where I am not having any problems. It's a meganode and that may speak to its capacity to handle large volumes of packets.

I was getting the same packet loss in NYC on Muliphein, but that suddenly resolved. Still, half of the six NYC meganodes are jittery, with lots of latency spikes.

I just rechecked everything. I also checked both the first and second hops off each VPN node to make sure that the packet loss is consistent and not localized to one node/router.

Can anyone at AirVPN figure out what's going on? I already talked to both my ISP and carrier about this and can completely rule out any problems with my connection physically, or at layers 2 or 3. See below for the latency and packet loss numbers.

Lacerta:

96 of 100 packets returned successfully : 4.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 146/0/75

Ross:

98 of 100 packets returned successfully : 2.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 166/0/58

Cephei:

95 of 100 packets returned successfully : 5.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 24/0/18

Kornephoros:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 32/0/18

Mintaka:

93 of 100 packets returned successfully : 7.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 76/0/21

Tejat:

86 of 100 packets returned successfully : 14.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 23/0/19

Tyl:

95 of 100 packets returned successfully : 5.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 24/0/18

Muliphein:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 262/0/31

Paikauhale:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 50/0/28

Sadalmelik:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 193/0/30

Terebellum:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 280/0/40

Unukalhai:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 44/0/28

Unurgunite:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 86/0/29

 

[Staff 10483]

@Baraka

Thank you.

After the private dialogue in the ticket you kindly opened with the support team we could manage to reduce the packet loss of Tyl and Tejat to 1% from/to the mentioned servers to/from the dozens of datacenter we operate servers in. Packet loss ≤ 1% is ideal for any application and purpose.

Can you confirm that the problem is solved in Tyl and Tejat?

We will proceed in the near future to investigate again about Mintaka, Lacerta and Cephei, where we could not reproduce the problem during our initial tests (we detected packet loss ≈ 0.0%).

Kind regards
 

[Baraka 33]

Tejat is still terrible, unfortunately. 100 quick pings showed 13% packet loss.

Tyl is better, but definitely over 1% in packet loss. My quick ping tally of 100 packets showed 5% packet loss. WinMTR showed 2%.

[Baraka 33]

I should add that Kornephoros is showing absolutely no packet loss, whatsoever. The jitter/latency variance is also very low. Rock solid node.

[Staff 10483]

@Baraka

Hello!

As you already know from the ticket, we could reproduce the problem only on Tyl with 5% packet loss from any source. We determined It was a momentary problem due to SYN flood. After all, the server was decently resilient to the attack showing only a limited packet loss during the flood. At the moment we see packet loss < 1% from 20 different countries and dozens of datacenters in the world toward any server including Tejat (0.0%), so we must consider the problem not reproducible at the moment.
 

12 hours ago, Baraka said:

I should add that Kornephoros is showing absolutely no packet loss, whatsoever.

Since 3 additional 10 Gbit/s servers were added recently in the very same datacenter where Kornephoros lives, you should test them as well:
https://airvpn.org/forums/topic/78643-three-new-10-gbits-servers-available-ca/

They offer you additional options on top of the several Canadian and USA servers which you already connect to without packet loss.

Kind regards
 

[Baraka 33]

I had no idea. This is great news! I've tested the new servers and sent them to my last exchange on my ticket. I'm cutting and pasting it below.

Castula:

93 of 100 packets returned successfully : 7.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 405/0/25

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              10.66.10.1 -   10 |   55 |   50 |   16 |  353 |  803 |  461 |
|     static-198-44-157-49.cust.tzulo.com -    7 |   59 |   55 |   17 |  384 |  803 |  462 |
|                   No response from host -  100 |   15 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   15 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   15 |    0 |    0 |    0 |    0 |    0 |
|                           192.178.99.39 -    7 |   59 |   55 |   17 |  344 |  803 |  461 |
|                          216.239.50.119 -    7 |   59 |   55 |   19 |  384 |  803 |  461 |
|                              dns.google -    7 |   59 |   55 |   17 |  383 |  803 |  461 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Chamukuy:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 30/0/18

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             10.68.138.1 -    0 |   51 |   51 |   16 |   32 |   91 |   25 |
|    static-198-44-157-129.cust.tzulo.com -    0 |   51 |   51 |   16 |   34 |  323 |   17 |
|                   No response from host -  100 |   11 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   11 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   11 |    0 |    0 |    0 |    0 |    0 |
|                          192.178.98.121 -    0 |   51 |   51 |   17 |   33 |  107 |   25 |
|                           142.251.68.25 -    0 |   51 |   51 |   17 |   32 |  108 |   25 |
|                              dns.google -    0 |   51 |   51 |   17 |   33 |   96 |   23 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Elgafar:

100 of 100 packets returned successfully : 0.0% PACKET LOSS 
Round Trip Time (in milliseconds) Maximum/ Minimum/Average: 20/0/17

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             10.78.138.1 -    0 |   50 |   50 |   16 |   46 | 1037 |   17 |
|     static-198-44-157-57.cust.tzulo.com -    0 |   50 |   50 |   17 |   47 | 1035 |   19 |
|                   No response from host -  100 |   10 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   10 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   10 |    0 |    0 |    0 |    0 |    0 |
|                           192.178.99.39 -    0 |   50 |   50 |   16 |   45 | 1027 |   17 |
|                          216.239.41.175 -    0 |   50 |   50 |   18 |   46 | 1026 |   19 |
|                              dns.google -    0 |   50 |   50 |   16 |   46 | 1034 |   22 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Kornephoros is like the last two and rarely disappoints, so I didn't test it. Castula is performing very badly. What do you think is going on??

[Staff 10483]

5 hours ago, Baraka said:

Kornephoros is like the last two and rarely disappoints, so I didn't test it. Castula is performing very badly. What do you think is going on??

Hello!

We think that the problem is on your side. Castula is absolutely perfect just like other servers you experience this problem on. We have no complaints whatsoever about any of the servers you mention.

Note that Castula, Chamukuy, and Elgafar are all connected to the same upstream in the same small subnet.

Your tests have been instrumental to make us aware of the problem (SYN flood and similar events) frequently occurring on specific Canadian servers, so thank you!

A good thing you can do on your side is black listing the servers that don't work well for you. You have anyway a vast range to pick from. Keep us informed if the problem suddenly appears on one or more of the servers that are perfectly fine for you now.

Kind regards