Alkes and Merope both down

[ableounceony 6]

Woke up this morning and both my regular servers seem to be down (Alkes and Merope) through both my pfSense box and via OpenVPN for Android. The Status page shows them as up, but the Ping Matrix just has blank lines for them.

 

EDIT:  Also, if I go to the "Checking Route" page, Alkes and Merope (as well as Sabik and Draco) all say nothing but "Pending, please refresh."

[nucshuco 0]

Same here. Hope it doesn't last long.

[ableounceony 6]

I switched my Merope (Los Angeles) IP address to the Aquila (Fremont) IP address in my pfSense setup and everything works fine, there.  It's definitely a problem with Alkes and Merope.  Not something local.

[DarkWrath 1]

I'm also having trouble connecting to any of the LA servers. I've raised a support ticket. I suggest everyone having issues do the same, so it can be resolved sooner rather than later.

 

Thanks.

[ableounceony 6]

I'm also having trouble connecting to any of the LA servers. I've raised a support ticket. I suggest everyone having issues do the same, so it can be resolved sooner rather than later.

 

Thanks.

 

I agree.  If anyone's having this issue, definitely open a support ticket (hopefully, the weekend won't stall things).

 

It's odd that the overall status page and the individual server status pages show no problems:

 

https://airvpn.org/status/

https://airvpn.org/servers/Alkes/

https://airvpn.org/servers/Merope/

 

And, according to the traffic graphs on those server status pages, both servers are up and many people are connected to them.  Yet, I can't get to those servers at all from here (Las Vegas) and the Checking Routes page and Ping Matrix both have nothing but blank lines for them:

 

https://airvpn.org/routes/

https://airvpn.org/pingmatrix/

 

The Latest Issues page does show some problems at both those servers about 4 hours ago, but nothing current:

 

https://airvpn.org/issues/

 

EDIT 1:  I don't know if this is even meaningful, but I can ping and do a tracert on the Merope server without problem:

 

PS C:\WINDOWS\system32> ping 199.241.146.162

 

Pinging 199.241.146.162 with 32 bytes of data:

Reply from 199.241.146.162: bytes=32 time=30ms TTL=53

Reply from 199.241.146.162: bytes=32 time=33ms TTL=53

Reply from 199.241.146.162: bytes=32 time=29ms TTL=53

Reply from 199.241.146.162: bytes=32 time=30ms TTL=53

 

Ping statistics for 199.241.146.162:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 29ms, Maximum = 33ms, Average = 30ms

PS C:\WINDOWS\system32> tracert 199.241.146.162

 

Tracing route to 199.241.146.162 over a maximum of 30 hops

 

  1    21 ms    22 ms    21 ms  10.8.0.1

  2    24 ms    21 ms    22 ms  ge0-1.core.quintex.com [199.249.223.254]

  3    32 ms    24 ms    26 ms  v1522.core2.fmt2.he.net [64.62.151.121]

  4    27 ms    22 ms    23 ms  10ge7-3.core3.fmt2.he.net [184.105.222.141]

  5    30 ms    22 ms    23 ms  10ge7-9.core1.sjc2.he.net [184.105.222.14]

  6    25 ms    23 ms    22 ms  pccwg-as-as3491.10gigabitethernet10-11.core1.sjc2.he.net [72.52.104.78]

  7    32 ms    31 ms    33 ms  TenGE0-0-0-0.br04.lax05.pccwbtn.net [63.223.43.2]

  8    31 ms    32 ms    31 ms  hugeserver.Te0-1-0-10.br04.lax05.pccwbtn.net [63.217.104.94]

  9    32 ms    34 ms    33 ms  199.244.116.2

 10    74 ms    33 ms    33 ms  199.244.116.38

 11    33 ms    37 ms    31 ms  199.241.146.162

 

Trace complete.

 

It's just that pfSense says I've got 100% packet loss to them and OpenVPN for Android says:

 

"read UDP [ECONNREFUSED]: Connection refused (code=111)"

 

EDIT 2:  According to what I can see in pfSense, the packet loss from those servers went to 100% at about 4:30 am 05/05/2018 (Pacific time) and stayed there.  That time agrees with the time on AirVPN's Issues page where they showed packet loss.  But, that page claims it was fixed about 15 minutes later.  Not here, though.

[diabolical 0]

So my guess is airvpn has colo with someone in 530 W. 6th st. That building is undergoing power maintenance today, and having been to that rat hole many times over the years it would be a huge surprise if this outage isn't related...

 

PLANNED UILITY MAINTENANCE - NOT SERVICE IMPACTING

Location: 530 W. 6th ST
Date: Saturday, May 5th 2018
Start Time: 04:00AM PDT
End Time: 09:00AM PDT
Expected Outage: None

Risk: Critical load will be transferred to standby emergency generators for the duration.

Description:
Los Angeles Department of Water & Power technicians will be performing one power vault service of one of the building's service vaults. Generator technicians will be on-site to observe the generators' status throughout the maintenance. On-site engineers and QuadraNet staff will transfer each affected floor to the generator, one by one, and ensure proper operation before proceeding. Once all relevant equipment is on generator power, the service vault will be shut down and maintenance will begin on the vault as well as busway connections and components.

After each vault's preventative maintenance is complete, power will be restored to the service vault and busway, and each affected floor will be transferred from generator to utility power. The maintenance will conclude when the generator is under no load, and is subsequently shutdown.

Electricians will be on-site in the unlikely event that any UPS or power system component does not transfer smoothly and requires service.

[Compuserve 0]

same thing here.

[Monotremata 22]

Ahhh should have checked here first haha. Just posted about it in the Eddie Beta thread because I thought it was the client doing it. Guess its just down for everybody.. 

[ableounceony 6]

Looks like it's back (Alkes, at least).  The Issues page says they had some "line issues" around 0100 hrs this morning (Sunday, Pacific time).