[US] Server replacements in LA, NYC, Phoenix

[Staff 10081]

Hello!

We're very glad to inform you that in the coming days, according to our infrastructure expansion and improvement plan, all the servers in Los Angeles, New York City and Phoenix will be replaced.

The new servers will have much more powerful hardware and all of them will be connected to a 10 Gbit/s (full duplex) port with guaranteed bandwidth. Therefore, the infrastructure in the mentioned locations will be entirely upgraded to new 10 Gbit/s servers, with 3 servers in LA, 3 servers in Phoenix and 4 servers in New York City. The current servers will be decommissioned on November the 1st.

Please note that all the new servers will have new IP addresses. Connectivity will be very similar, with the notable addition of GTT and Zayo as fiber and tier1 transit providers.

Kind regards and datalove
AirVPN Staff
 

[cccthats3cs 14]

I greatly appreciate this announcement and change. However, I have a few questions regarding specific points. If any are unable to be answered that is fine, but regarding the actual servers and network, I am in favor of as much transparency as possible.

1. Is the infrastructure expansion and improvement plan available anywhere, and if not, is it possible to post such plans on a regular basis as it changes?
2. All of the servers in those existing locations are hosted by M247. Will these new servers also be hosted in M247 datacenters?
3. What are the current transit providers in each location before the addition of GTT and Zayo? (This is where my networking experience is not enough, because I thought as each server had M247 IP addresses, that the connectivity would be identical, i.e., same routing, transit providers, and so on comparing an AirVPN New York server at M247 to any competitor New York VPN server at M247.)
4. There were previous reports that M247 falsified or otherwise changed the Phoenix server location and that the true location is Los Angeles. Has M247 (or if not, the new datacenter) confirmed that the new servers are to be provisioned in a Phoenix datacenter? The current BGP whois for the Bootes server /24 shows references to Phoenix and a Phoenix datacenter (https://bgp.he.net/net/193.37.254.0/24#_whois), but this is refuted by pings obtained from Phoenix and Los Angeles looking glass servers.

Thanks

[go558a83nk 367]

10 hours ago, cccthats3cs said:

I greatly appreciate this announcement and change. However, I have a few questions regarding specific points. If any are unable to be answered that is fine, but regarding the actual servers and network, I am in favor of as much transparency as possible.

1. Is the infrastructure expansion and improvement plan available anywhere, and if not, is it possible to post such plans on a regular basis as it changes?
2. All of the servers in those existing locations are hosted by M247. Will these new servers also be hosted in M247 datacenters?
3. What are the current transit providers in each location before the addition of GTT and Zayo? (This is where my networking experience is not enough, because I thought as each server had M247 IP addresses, that the connectivity would be identical, i.e., same routing, transit providers, and so on comparing an AirVPN New York server at M247 to any competitor New York VPN server at M247.)
4. There were previous reports that M247 falsified or otherwise changed the Phoenix server location and that the true location is Los Angeles. Has M247 (or if not, the new datacenter) confirmed that the new servers are to be provisioned in a Phoenix datacenter? The current BGP whois for the Bootes server /24 shows references to Phoenix and a Phoenix datacenter (https://bgp.he.net/net/193.37.254.0/24#_whois), but this is refuted by pings obtained from Phoenix and Los Angeles looking glass servers.

Thanks

The phoenix location is definitely spoofed.  You can check the m247 site to see their list (or a map) of their actual locations.  The Phoenix location as it is adds bandwidth but not route diversity since it's likely found by the same route as the LA location.  Something better may be out there but maybe not.  https://m247.com/eu/data-center-locations/

[Staff 10081]

14 hours ago, cccthats3cs said:

1. Is the infrastructure expansion and improvement plan available anywhere, and if not, is it possible to post such plans on a regular basis as it changes?

Hello!

This information will not be disclosed at the moment in details but you may be informed in due time.
 

14 hours ago, cccthats3cs said:

2. All of the servers in those existing locations are hosted by M247. Will these new servers also be hosted in M247 datacenters?

No to the question and no to the assumption (wrong assumption). All of the current servers are in M247 property racks but not necessarily in M247 datacenters. The new servers will not necessarily be in the same datacenters and M247 will not own any of the new hardware.
 

3 hours ago, go558a83nk said:

3. What are the current transit providers in each location before the addition of GTT and Zayo?

The current ones are M247, Cogent and Hurricane.
 

3 hours ago, go558a83nk said:

4. There were previous reports that M247 falsified or otherwise changed the Phoenix server location and that the true location is Los Angeles.

Those reports spread false information, if they exist. The current Phoenix servers are in a Cogent datacenter located in Phoenix, this one: https://www.datacentermap.com/usa/arizona/phoenix/cogent-phoenix/
New servers are in a Phoenix datacenter as well.

Kind regards
 

[Staff 10081]

3 hours ago, go558a83nk said:

The phoenix location is definitely spoofed.  You can check the m247 site to see their list (or a map) of their actual locations. 

Hello!

The current Phoenix servers are in a Cogent datacenter in Phoenix, see also our previous message. Any provider may offer services inside totally owned or partially owned datacenters, or from not owned at all datacenters where entire racks are rented or similar agreements are finalized.

The new Phoenix servers are in Phoenix, shipped at the beginning of October and arrived on site a few days ago, they are already connected and working, we are testing them and we are confident we will announce them before the end of October. There are dozens of datacenters in Phoenix, around 75 or so, including big ones like Google, Apple, NTT, Cogent and more.

Kind regards
 

[vpntest012 1]

This is indeed great news!  Well done AirVPN!
 

On 10/26/2024 at 1:00 AM, Staff said:

We're very glad to inform you that in the coming days, according to our infrastructure expansion and improvement plan, all the servers in Los Angeles, New York City and Phoenix will be replaced.

Does it include Saclateni, which was recently deployed late of last year?  If so, are the new servers more powerful than Saclateni?  And will each of these servers have full-duplex bandwidth of 20 Gbps?
 

On 10/27/2024 at 2:26 AM, Staff said:

The current Phoenix servers are in a Cogent datacenter located in Phoenix, this one: https://www.datacentermap.com/usa/arizona/phoenix/cogent-phoenix/

As you can see from the attached screenshot, connecting to Saclateni from Anaheim, CA, it shows similar latency (~ 8-9 ms) to other servers (Groombridge and Teegarden) in Los Angeles, CA.  However, it's always puzzled me why I'd get similarly low latency (~ 8-9 ms) to other servers (Bootes, Chalawan, Indus, and Virgo) in Phoenix, AZ.  How is it possible?  It defies the laws of physics!  Anaheim, CA to Los Angeles, CA is a shorter distance than to Phoenix, AZ.  And latency is definitely affected by distance.  Unless these ISPs figure out a way to tunnel packets through a worm hole!  I call shenanigans! 😁
 

Edited ... by vpntest012
Some questions are answered; formatting

[ersatzzz 5]

I usually connect to my closest servers, which are NYC (Haedus, Ikill. etc). However as of today those servers are no longer available. I've tried to connect to each of the four "new" servers in NYC, but the Eddie client in macOS never connects. Is there something special one needs to do to connect to the new servers via Eddie?

[Staff 10081]

8 hours ago, ersatzzz said:

I usually connect to my closest servers, which are NYC (Haedus, Ikill. etc). However as of today those servers are no longer available. I've tried to connect to each of the four "new" servers in NYC, but the Eddie client in macOS never connects. Is there something special one needs to do to connect to the new servers via Eddie?

Hello!

No, nothing special. The configuration is identical. Does the system report tell anything useful?

Kind regards
 

[Stalinium 52]

On 10/31/2024 at 2:12 AM, vpntest012 said:

How is it possible?  It defies the laws of physics!

This must not always be impossible, sometimes the routing is really busted. For example, my ISP routes even intra-city traffic through their big PoP (point of presence) at the IX (internet exchange) which is ~20ms away.
That said, I have similar questions about the Madrid and Berlin servers by M247. PS: I had not checked them again since Edited ... by Stalinium
Add PS

[vpntest012 1]

Hello,

What is going with the Tzulo servers and/or connections in Los Angeles and Phoenix?  There have been extended high packet loss for weeks.  Also, the upload speed have been under performing, usually 150 - 300 Mbps (for symmetrical Gigabit fiber) for the last 2 - 3 weeks as well.

Hope these issues get fixed soon!

Thanks,

[Air4141841 30]

I can only imagine what COULD be affecting Los Angeles right now....
 

[Staff 10081]

5 hours ago, vpntest012 said:

What is going with the Tzulo servers and/or connections in Los Angeles and Phoenix?  There have been extended high packet loss for weeks. 

Hello!

According to the datacenters, this is a consequence of heavy flood attacks which compel the IP transit provider to null route specific IP addresses. We are assessing the situation.

Kind regards
 

[vpntest012 1]

On 1/10/2025 at 8:47 PM, vpntest012 said:

Hello,

What is going with the Tzulo servers and/or connections in Los Angeles and Phoenix?  There have been extended high packet loss for weeks.  Also, the upload speed have been under performing, usually 150 - 300 Mbps (for symmetrical Gigabit fiber) for the last 2 - 3 weeks as well.

Hope these issues get fixed soon!

Thanks,

Ten days later, and the upload speed is still really low.  It is not like the servers are busy (max 25% utilization) at non-peak time, 5:00 AM local time.

[oassQ9w4cbl4AySZhhth%p36x 4]

On 10/31/2024 at 1:12 AM, vpntest012 said:

This is indeed great news!  Well done AirVPN!
 

Does it include Saclateni, which was recently deployed late of last year?  If so, are the new servers more powerful than Saclateni?  And will each of these servers have full-duplex bandwidth of 20 Gbps?
 
 As you can see from the attached screenshot, connecting to Saclateni from Anaheim, CA, it shows similar latency (~ 8-9 ms) to other servers (Groombridge and Teegarden) in Los Angeles, CA.  However, it's always puzzled me why I'd get similarly low latency (~ 8-9 ms) to other servers (Bootes, Chalawan, Indus, and Virgo) in Phoenix, AZ.  How is it possible?  It defies the laws of physics!  Anaheim, CA to Los Angeles, CA is a shorter distance than to Phoenix, AZ.  And latency is definitely affected by distance.  Unless these ISPs figure out a way to tunnel packets through a worm hole!  I call shenanigans! 😁
  

hi this is a common question that most people inevitably fall foul of because they don't fully understand how routing and the internet in general works. (its also something i fell foul of too and took a long time to learn)

the internet itself is made up of millions of different peering agreements, all of which you are never privy to and all have various limits imposed. in addition to that, you also have physical hardware limits as well as rate limiting involved there too. 

so you can end up in weird scenarios, where you get faster speeds and lower latency connecting to geographically areas. this doesn't defy the laws of physics, its just the result of rules that you are not able to see. 
in my case for example with my previous ISPs during london prime time, it was faster for me to connect to an NL node and double back on myself (the route effectively being HOME UK > NL > UK) than it was to connect from my home to the uk vpn node. this was because of traffic limits surrounding intra-london / uk traffic. 

it was a big difference too, we are talking 30ms and 80mbps speeds with the 'shortest' / most efficient path, and the longer one being 24ms and 500mbps. this was all due to the nature of these agreements, and various networking traffic management techniques that are applied. by going outside of the country and looping back, i ended up being governed by a different set of rules, which resulted in better speeds. 

remember also that 99% of all ISPs are not looking for the least latency or fastest throughput for you, instead they are looking for the cheapest bulk data transfer. cogent specifically is one of those providers which are known to be quite cheap for bulk transit, moreso than other providers. 

you as the home user have little control over all of this, the best thing you can do is try find something which is a fastpath for you and stick to it. if none of them are, then you can roll your own vps and be happy with that instead, or use it as an intermediary fast path and chain vpn's together. something like HOME > VPS > VPN. 

this is a long process to find this kind of information out, as its not easily accessible. to help with it you can check peeringdb or similar sites which show who your ISP peers with, and try pair matching. that at least ensures that you have some route and you can determine capacity, that doesn't necessarily mean it will be fast though. you need to dig deeper and map things out. its a pain in the ass but probably won't take more than a day to do it all and then also research various VPS / VPN nodes. remember to do this kind of testing during peak times too because those are going to be the best (worst performance) examples of real world usage. 

[kbps 29]

1 hour ago, oassQ9w4cbl4AySZhhth%p36x said:

so you can end up in weird scenarios, where you get faster speeds and lower latency connecting to geographically areas. this doesn't defy the laws of physics, its just the result of rules that you are not able to see. 
in my case for example with my previous ISPs during london prime time, it was faster for me to connect to an NL node and double back on myself (the route effectively being HOME UK > NL > UK) than it was to connect from my home to the uk vpn node. this was because of traffic limits surrounding intra-london / uk traffic. 

it was a big difference too, we are talking 30ms and 80mbps speeds with the 'shortest' / most efficient path, and the longer one being 24ms and 500mbps. this was all due to the nature of these agreements, and various networking traffic management techniques that are applied. by going outside of the country and looping back, i ended up being governed by a different set of rules, which resulted in better speeds. 
 

Can confirm this.  I had issues at peak time in the UK with UK servers but was fine when using NL.

https://airvpn.org/forums/topic/57097-all-uk-servers-slow-downloadupload/

For a long while I used NL servers.  However traffic is now good at peak times so I am back to the UK.  As you say, we have little control on what goes on behind the scenes.