Hyonix - Chicago (CHI1) Datacenter Outage – Incident details

All systems operational

Chicago (CHI1) Datacenter Outage

Resolved
Major outage
Started about 1 month agoLasted about 1 hour

Affected

Virtualization Hosts

Major outage from 3:04 PM to 4:25 PM

Chicago, Illinois (CHI1)

Major outage from 3:04 PM to 4:25 PM

Core Networks

Major outage from 3:04 PM to 4:25 PM

Chicago, Illinois (CHI1)

Major outage from 3:04 PM to 4:25 PM

Updates
  • Resolved
    Resolved

    Our team was able to confirm that all services have been back online at this time.

    If you still having issues with your server, please open a ticket.

    We've already had redundancy in place for cases like this, specifically in this case, our Core1 router went down due to power losses from the Datacenter. Leaving Core2 online with our backup carrier.

    However, this backup carrier (Cogent) did not pick up our routes properly, causing our network to go completely down, we will be looking to replace this backup carrier in the coming weeks to prevent this from happening again in the future.

    The cabinet started to go online once the Datacenter fixed the power issue, making our Core1 router back online. Most cabinets were not affected by this power losses, only our core cabinets were affected.

    We apologies for any inconvenience caused.

    Please let us know if you have any further questions on this incident!

  • Update
    Update

    Our team is getting network back online after the power reset affecting our core switches. Routes are being learnt and populated. We will keep you updated.

  • Identified
    Identified

    Coresite has confirmed a power issue in our cabinets and has been working to resolve it.

    Our monitors alerted us that networks are being restored at this time, our network admin team is double checking at this time.

  • Update
    Update

    Our technician is currently en-route to the DC but will be slightly delayed due to heavy traffic in downtown Chicago.


    Our team is currently still waiting for responses from the Datacenter, they have not provided any updates on our case.

  • Update
    Update

    Our team is still waiting for Datacenter (Coresite CH2) Response on this matter. They have acknowledged immediately but we have not received any further information from them yet.

    Our last log indicate that there is power losses in the Datacenter, our team access is completely cut as none of our devices are reachable.

  • Investigating
    Investigating

    Our team is aware of issue that causing unavailability at our CHI1 Datacenter. We are currently investigating this incident.