Hyonix - NYC2 Outage – Incident details

All systems operational

NYC2 Outage

Resolved
Partial outage
Started about 1 year agoLasted 7 days

Affected

Virtualization Hosts

Partial outage from 2:53 AM to 4:35 AM, Degraded performance from 4:35 AM to 6:42 AM, Operational from 6:42 AM to 6:28 PM

New York, New York (NYC2)

Partial outage from 2:53 AM to 4:35 AM, Degraded performance from 4:35 AM to 6:42 AM, Operational from 6:42 AM to 6:28 PM

Core Networks

Partial outage from 2:53 AM to 4:35 AM, Degraded performance from 4:35 AM to 6:42 AM, Operational from 6:42 AM to 6:28 PM

New York, New York (NYC1)

Partial outage from 2:53 AM to 4:35 AM, Degraded performance from 4:35 AM to 6:42 AM, Operational from 6:42 AM to 6:28 PM

Updates
  • Resolved
    Resolved

    This incident has been marked as resolved as there has been no further issues after 7 days!

  • Monitoring
    Monitoring

    The Datacenter Operators have confirmed that maintenance has been completed, there is no further works being done, all powers have been restored.

    Our team will still monitor the situation for 1-2 days before closing this incident.

    We apologizes for any issues or inconvenience may have caused.

    Thank you for your patience and understanding during this process.

    Please let us know via ticket if you still have any issues!

  • Update
    Update

    We've completed assessment on this incident.


    Root cause: The Datacenter is performing maintenance related to electrical system upgrade. During the maintenance window, the power circuits of the cabinet row hosting our main network cabinet has tripped, causing our routers to lose both main and redundant power sources, leading to all other cabinets losing network access.


    Solution: Upon arrival, our technician noticed that many electricians were aware of the problem and were already actively working on it. The power was restored shortly after, however due to the power getting completely tripped, a few of our cabinets were affected. Our tech team started to go over every cabinet to ensure that all servers were brought online.


    Current state: All servers are back online and operational at this time. The Datacenter is still under maintenance but don't expect any further issues. We were given ETA for completion to be at 09/18/2024 04:00:00 am EDT. Our team will update once we heard confirmation from the Datacenter.

    Customers with servers restarted due to the power loss will be notified via ticket. Our team will be monitoring the issue closely.

    Our apologies for any inconvenience caused.

    Please reach out to us via ticket if you still having issues with your servers.

  • Update
    Update

    Datacenter Operators confirmed the power outage in a section of the Datacenter hall. Affecting many rows of cabinets. Electricians are also on site and bringing the power back up.

    Our team is currently assessing the impact of the power outages and will post further update shortly.

  • Identified
    Identified

    Our technician is currently on-site at this time, it appears the Datacenter is suffering a major power outages. we are still checking with Datacenter Operators on the root cause.

    More update to follow!

  • Investigating
    Investigating

    Our team is aware of an issue related to our NYC2 Datacenter being down. NYC1 still remains fully operational.

    The issue started at: 02:27 AM UTC. We've immediately dispatched our technician to the Datacenter to check on the status.

    We are looking into this issue at this time!

    Our team will post updates as we receive news from the Datacenter.