2020-04-08 Network Outage

parsonswy

10:07am EST
We're aware of a network outage affecting access to most of our servers. The issue is with an upstream ISP called 'NTT America' so once again we have to wait for them to resolve the issue.
We will continue to monitor the situation and invite you to check our live monitoring data as well

General Network Status
Network Specific Metrics

parsonswy

10:30am EST
We've been told that someone has initiated a DDoS attack against the IP range in which some of our servers reside. The scope of the attack requires IP routing changes in order to mitigate.

parsonswy

10:50am
Mitigation is in place and the network has stabilized. We've been told to expect a brief outage when the route changes are reverted to disable filtering after the attack has ceased. It should not be as noticeable as the initial outage.

parsonswy

11:58am EST
The actors seem to have altered their attack strategy. PL and latency are up again, but we expect the mitigation provider to compensate shorty.

parsonswy

12:10pm EST
PL is nominal and latency is down, though it seems the mitigation strategy requires sending traffic from Virgina -> Los Angles -> Virginia -> Dallas which is _100ms of just travel time so expect a slight, but consistent delay in actions until further notice.

parsonswy

7:21pm EST
We observed further routing changes and traffic is no longer being sent to LA, but is still being filtered regionally in VA. PL is nominal, RTT is nominal.

parsonswy

11:15pm EST
Situation unchanged. Metrics are mostly nominal with some spikes. Mitigation is still in effect - mtr actually shows DDOS-FILTER on 11th hop from nva0.pcn to nva1.pcn.