Hyonix - Notice history

This incident has been marked as resolved as there was no further issues.

After extended troubleshooting with all parties, we've identified that the issue is caused by a faulty cross-connect from Datacenter side. The cross-connect has been replaced at this time and our network is now back at full capacity. We will continue to monitor this issue for another 48 hours before closing.

If you have any further questions about this incident, please let us know via ticket!

Thank you for your patience and understanding.

Currently our team is engaging Zayo technician but we still haven't been able to identify the root cause yet. We are suspecting an cross-connects fault on either Datacenters. Once Zayo has fully tested the fiber and the issue still not resolved, our team will start engaging with both Datacenters to troubleshoot on-site cross-connects.

Our team found out a bottleneck that caused slow deployment, this has been on-going randomly for at least 3 months and finally became more apparent that our team can replicate and trace to the root cause.

This has now been fixed, all orders are now being processed as normal and faster than before!

We highly appreciate your patience and understanding during this process. Please do let us know via ticket if you still need any further assistance!

Our team has received the final Reason for Outage (RFO) from the datacenter and has summarized key details below. Our team is also discussing internally various options that can enhance service reliability and prevent similar incidents in the future.

We sincerely apologize for the service disruption and any inconvenience this incident may have caused your operations. We fully understand the impact such outages can have on your business and take this matter very seriously.

If you have any questions about this incident or need additional information for your records, please don't hesitate to contact us via support ticket.

----

Data Center Power Outage - May 6, 2025

West 7 Center - Data Halls L1-130 & L1-135

Incident Overview

On May 6, 2025, West Center experienced a critical power and cooling outage affecting data halls L1-130 and L1-135. The incident began at 3:45 PM and was fully resolved by 9:00 PM, resulting in approximately 5 hours and 15 minutes of service outage.

Detailed Timeline of Events

3:45 PM - Initial Detection

• Smoke alarm activated in the main power switchgear room

• Building engineering team responded immediately to investigate the source

• Backup generators automatically started due to utility power loss

• Fire safety protocols took priority, preventing immediate generator troubleshooting

4:15 PM - Smoke Alarm Reset

• After 45 minutes of investigation, the smoke alarm was reset within the Fire Life Safety system

• The source of smoke remained undetermined at this time

• Engineering team suspected the smoke originated from the utility company's transformer area (inaccessible to facility staff)

4:15 PM - 4:30 PM - Power Investigation

• Engineers began investigating the cause of generator activation

• Multiple potential triggers required systematic elimination to identify the root cause

4:30 PM - UPS System Failure

• The Uninterruptible Power Supply (UPS) system in L1-135 shut down due to battery depletion

• The UPS had been cycling between battery and unstable generator power, preventing proper load transfer

• Generator instability caused the system to enter "Lock-Out" condition

5:20 PM - Root Cause Identified

• Engineers located the source: a tripped 4000-amp utility breaker in the main switchgear room

• This breaker supplied power to both affected data halls

5:33 PM - Utility Power Restored

• The 4000-amp utility breaker was successfully reset

• Computer Room Air Handlers (CRAH) units in L1-135 automatically restarted

• UPS system in L1-135 failed to restart automatically

7:45 PM - UPS Technician Arrival

• Specialist UPS technician arrived on-site

• Discovered that the input breaker to the UPS Static Bypass Module had tripped due to intermittent ground fault conditions

8:30 PM - UPS System Restart

• UPS technician successfully reset the tripped breaker and restarted the UPS system

9:00 PM - Full Service Restoration

• All systems in both L1-130 and L1-135 returned to normal operation

Root Cause Analysis

Initial Investigation (May 2025) The facility's engineering team, along with senior electrical consultants from Fakouri Electrical Engineering and Ramboll Electrical Engineering Group, conducted comprehensive testing of all electrical infrastructure. The investigation revealed that the incident was caused by an intermittent ground fault originating from customer equipment - specifically a faulty power distribution unit (PDU) located within a customer cabinet in L1-130.

Updated Root Cause (May 26, 2025) Further investigation revealed the complete picture: the ground fault from the customer equipment triggered an arc flash in the 4000-amp bus duct located in the ceiling of the main switchgear room. This arc flash caused an explosion that created a puncture approximately six inches from the adjacent wall, making it difficult to detect during initial inspections. The damaged bus duct connected the utility feed to the affected data halls.

Corrective Actions Taken

Immediate Actions:

• May 14, 2025: Removed the identified faulty customer PDU from L1-130

• Identified and began monitoring a second cabinet in L1-130 that was activated less than 12 hours before the incident

• Scheduled additional inspection with third-party consultant (Hawthorne Power System)

Infrastructure Repair:

• May 22, 2025: Completed emergency replacement of the damaged 4000-amp bus duct

• Collaborated with three engineering consultants and preferred electrical contractors

• Successfully rerouted power to backup utility source during replacement

• Replacement completed without service interruption (7:00 AM - 2:30 PM)

Policy Changes:

• Implemented new mandatory testing and inspection policy for all customer-provided equipment

• All customer equipment must now undergo facility engineering review before installation

• Enhanced monitoring procedures for recently installed customer equipment

Preventive Measures

The data center has implemented several measures to prevent similar incidents:

1. Equipment Inspection Protocol: All customer-provided equipment must undergo comprehensive testing and inspection before installation and use within the facility

2. Enhanced Monitoring: Increased surveillance of customer equipment, particularly newly installed systems

3. Third-Party Validation: Engaging additional senior power equipment consultants for independent facility assessments

4. Infrastructure Improvements: Complete replacement of damaged electrical infrastructure with new, properly tested components

Current Status

All electrical infrastructure has been fully restored and upgraded. The facility is operating normally with enhanced safety protocols in place. The emergency bus duct replacement was completed without any service interruption, and all systems have undergone comprehensive testing to ensure reliability.

The data center management has emphasized that this incident represents an unusual and isolated occurrence in their three decades of critical facility operations, and they remain committed to maintaining the highest standards of reliability and safety.

It appears that due to the recent power outages, one of our network optics to the main network were affected and has became faulty causing the entire connection to constantly flaps, due to the link keeps flapping and not going down entirely, the auto fallback procedure did not kick in.

We've now disabled this faulty port and re-enabled our main network. Everything remains stable at this time. Our technician is scheduled to go to the Datacenter to perform replacement of the network optics by end of tomorrow, no further network impact is expected on our end at this time.

Our apologies for any inconvenience caused. We understand that the recent incidents have raised concerns about our service quality. Please rest assured that we are working actively internally to make sure these kind of incidents to not resurface again.

Our team is still pending Reason for outage (RFO) from the Datacenter in the recent power outage and steps taken to prevent this from happening again as they mentioned that they are conducting a throughout review before issuing an official statement. We will post update here once we hear back from them

Thank you for your patience and understanding on this matter!

We've been able to narrow this down to network-only event and not power event. Our team is looking into this.

We are observing very high packet losses toward our network, suspecting transport fault to our core network location.

Our team is forcing fall back to backup network at this time. Elevated latency is expected until we restore the main network.

It appears that our infrastructure is down again just now. We are contacting Datacenter for more information at this time.

Further updates will follow as we receive information from Datacenter staff.

Our apologies for any inconvenience caused. Please rest assured that our team is actively working on this issue at the moment.

Power and network have been restored completely. Our team will keep this incident open until we receive an official update from the datacenter.

Datacenter electrical engineers are on-site and working to restore power to racks.

Uninterruptible Power Supply (UPS) could not be turned on due to the fire alarms went off.

There is no official estimation as to when this will be completed yet. Our team is still on-site and actively working to get more information at this time.

Our in-house technician has arrived on site and confirmed that all our devices are not affected. We still haven't heard back from Datacenter site team about estimated time of restoration at this time.

Our team is still working actively on this and will provide more updates soon!

The Datacenter has confirmed that the fire alarm went off, which is why all power within the suite was cut.

Currently, the power is being restored to the suite as the site team is clearing the alarm and assessing the damages. However, it will take time to restore power to the racks. We are still trying to get an estimation from the Datacenter on when our racks can be restored.

We will provide more updates as we receive more information from the Datacenter! Thank you for your patience and understanding on this issue.

We've received notification from the Datacenter that the entire suite is experiencing power outages. Datacenter operator is aware of this situation and is investigating into it.

Our in-house technician has been dispatched on site at this time. Estimated 30-40 minutes for arrival.

We are still actively working with the Datacenter on this issue and will provide updates as more information is available to us.

Our apologies for any inconvenience caused.

There has been no further errors since the replacement. Our team will close this incident at this time as all services are online and stable.

Please let us know via ticket if you still have any issues!

We've completed the replacement, all connections are now back online and stable. Our team is currently monitoring these connections to see if any further issues.

All services are confirmed to be online, no further impact is expected at this time.

Currently we are attempting to replace the cross-connect fiber at this time. Disruptions is expected due to current connection constantly throwing errors.

Service is currently up, but we are expecting that it might go down again since our team is seeing many errors toward this cabinet connections. We are still working actively with the datacenter to clarify on the issue.

All services are back online at this time. We are still working with Datacenter technician for clarifications.

Our technician and datacenter technicians are at the cabinet troubleshooting the issue, first look it doesn't appear to be power issue, all connection between our cabinet and our core networks were not connecting. We are currently escalating to Datacenter intra-datacenter cross connect department to confirm the issue.

Currently local technician has been dispatched to the site to physically investigate on the issue, estimated time of arrival is 30 minutes.

We've experienced another attack flood, our team is actively mitigating at this moment.

All services remains online with a slight network disruption, observed maximum of 3 minutes.

We will continue to monitor and will post update as we receive more information.

We've experienced a network attack which exceeded our local capacity and brought the link down. Our team has been aware and started to mitigate as soon as we receive alerts.

As of now, all services are fully operational, we are currently monitoring the situation and will provide feedback as we have more information. We've observed network disruptions up to 6 minutes.

Our apologies for any inconvenience caused.

Please let us know via ticket if you still experiencing any issues.