Hyonix - Network outage in Virginia location – Incident details

We've completed all replacements at this time and re-established our main uplinks, all servers are now stable (no further disruption is expected) with maximum capacity and shouldn't have any downtimes from now on.

Thank you for your patience and understanding throughout this issue.

If you have any questions, please let us know via ticket!

Our team is currently working on new configuration for the new switches, your servers might be disrupted for ~1-2 minutes during this time as we transfer loads in between.

We will be replacing all backbone switches (even though just 1 was affected) with a new model to prevent this from happening in the future. However, we expect that all servers will remain fully operational during this time (except a few hiccups as mentioned), our team will do our best to have this completed ASAP.

The current ETA for completion is 4-8 hours.

Our team has replaced the hardware (backbone switches), we are working on setting up switches remotely and waiting for our upstream to update/verify their configuration to transfer the loads over.

ETA for completion: 4-7 hours.

We've been able to narrow down the issue to our backbone switch, our technicians are replacing switches on both Datacenters (IAD1 and IAD2) in order to ensure that issues won't happen again.

ETA for completion: 3 hours.

Our technician has arrived at the Datacenter and getting ready to perform troubleshooting.

Our team has been aware that there was another short downtime as our main link had gone down (again), but our backup link has picked up the loads with slight delays as the main link was not down right away but kept flapping.

We've scheduled an on-site technician to check out the issues earlier today (when the issue initially started) and it's expected that our technicians will arrive at the datacenter in 2 hours to troubleshoot the issue more throughout as there are things that we can't check remotely.

All servers are back online at this time. Our team will post further updates as we receive new information.

Thank you for your patience and understanding on this matter!

We've been able to re-established connection with our main upstream and all servers should be back normal!

Our team will be monitoring the issue closely, we will look into why the backup link did not work and will make appropriate adjustment to make sure this won't happen again.

We sincerely apologize for the inconvenience caused.

If you have any questions, please let us know via ticket!

We've found out that the fiber path between our network and our main upstream is not working. Our team is currently reaching out to related parties for resolution.

Our network has been transferred to our backup link (Cogent), however, it is still not reachable. We are actively working on this.