Hyonix - Notice history

Our team was able to confirm that all services have been back online at this time.

If you still having issues with your server, please open a ticket.

We've already had redundancy in place for cases like this, specifically in this case, our Core1 router went down due to power losses from the Datacenter. Leaving Core2 online with our backup carrier.

However, this backup carrier (Cogent) did not pick up our routes properly, causing our network to go completely down, we will be looking to replace this backup carrier in the coming weeks to prevent this from happening again in the future.

The cabinet started to go online once the Datacenter fixed the power issue, making our Core1 router back online. Most cabinets were not affected by this power losses, only our core cabinets were affected.

We apologies for any inconvenience caused.

Please let us know if you have any further questions on this incident!

Our team is getting network back online after the power reset affecting our core switches. Routes are being learnt and populated. We will keep you updated.

Coresite has confirmed a power issue in our cabinets and has been working to resolve it.

Our monitors alerted us that networks are being restored at this time, our network admin team is double checking at this time.

Our technician is currently en-route to the DC but will be slightly delayed due to heavy traffic in downtown Chicago.

Our team is currently still waiting for responses from the Datacenter, they have not provided any updates on our case.

Our team is still waiting for Datacenter (Coresite CH2) Response on this matter. They have acknowledged immediately but we have not received any further information from them yet.

Our last log indicate that there is power losses in the Datacenter, our team access is completely cut as none of our devices are reachable.

Both side of powers have been restored at the Datacenter, all Asus servers have been re-installed with redundant PSU at this time.

We will now close this status as the issue is resolved.

Please let us know via ticket if you still have any issues!

We have temporarily detached the failed PSU as a workaround to this issue, the CPU has stopped throttling and all servers are reachable at this time.

Please note that the servers were not down, just the performance of the servers were heavily degraded.

Our team has reported this to Asus engineer and they are aware of the issue.

We will keep monitoring the situation until the Datacenter power fully restored.

Our apologize for any inconvenience caused!

Please let us know via ticket if you still having issues.

Our technician has arrived on-site and is checking on the issue.

This incident has been marked as resolved as there has been no further issues after 7 days!

The Datacenter Operators have confirmed that maintenance has been completed, there is no further works being done, all powers have been restored.

Our team will still monitor the situation for 1-2 days before closing this incident.

We apologizes for any issues or inconvenience may have caused.

Thank you for your patience and understanding during this process.

Please let us know via ticket if you still have any issues!

We've completed assessment on this incident.

Root cause: The Datacenter is performing maintenance related to electrical system upgrade. During the maintenance window, the power circuits of the cabinet row hosting our main network cabinet has tripped, causing our routers to lose both main and redundant power sources, leading to all other cabinets losing network access.

Solution: Upon arrival, our technician noticed that many electricians were aware of the problem and were already actively working on it. The power was restored shortly after, however due to the power getting completely tripped, a few of our cabinets were affected. Our tech team started to go over every cabinet to ensure that all servers were brought online.

Current state: All servers are back online and operational at this time. The Datacenter is still under maintenance but don't expect any further issues. We were given ETA for completion to be at 09/18/2024 04:00:00 am EDT. Our team will update once we heard confirmation from the Datacenter.

Customers with servers restarted due to the power loss will be notified via ticket. Our team will be monitoring the issue closely.

Our apologies for any inconvenience caused.

Please reach out to us via ticket if you still having issues with your servers.

Datacenter Operators confirmed the power outage in a section of the Datacenter hall. Affecting many rows of cabinets. Electricians are also on site and bringing the power back up.

Our team is currently assessing the impact of the power outages and will post further update shortly.

Our technician is currently on-site at this time, it appears the Datacenter is suffering a major power outages. we are still checking with Datacenter Operators on the root cause.

More update to follow!