Hyonix - Notice history

All systems operational

Operational

Hyonix Panel - Operational

Hyonix Backend - Operational

Hyonix API - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Notice history

Dec 2024

Network ASN Migration
  • Completed
    December 07, 2024 at 10:55 PM
    Completed
    December 07, 2024 at 10:55 PM

    Maintenance has now been completed!

  • Planned
    December 07, 2024 at 6:45 PM
    Planned
    December 07, 2024 at 6:45 PM

    We will be performing a maintenance on our network by changing ASN 213122 to AS931 to consolidate network and enhance performance.

    Our team do not expect any issues from this change.

  • In progress
    December 07, 2024 at 6:45 PM
    In progress
    December 07, 2024 at 6:45 PM

    We will be working on LAX1, NYC1, MIA1, DAL1 location in this time schedule. Maintenance has been started.

Nov 2024

Adding new carrier in London
  • Completed
    November 30, 2024 at 6:07 PM
    Completed
    November 30, 2024 at 6:07 PM

    Maintenance has been completed! Network has been normalized. We do not expect any further interruptions at this time.

    Our team has successfully added another 2x 100Gbps network uplink with more direct connections toward major networks at this time.

    We are looking to perform even more upgrades in the next 2 weeks. Rest assured that upgrades will only be done in weekend to prevent disruptions to customers usage.

    If you still experience any issues in London, please reach out to us via ticket.

    Thank you for your patience and understanding!

  • In progress
    November 30, 2024 at 2:02 PM
    In progress
    November 30, 2024 at 2:02 PM

    Maintenance is now in progress.

  • Planned
    November 30, 2024 at 2:00 PM
    Planned
    November 30, 2024 at 2:00 PM

    We are adding new network PoP and bringing in new upstreams that would provide more redundancy and enhance performance further!

    Some minor network disturbance is expected at this time as we shift network loads to the new upstream once online!

    If you have any questions, please open a new ticket.

Chicago (CHI1) Datacenter Outage
  • Resolved
    Resolved

    Our team was able to confirm that all services have been back online at this time.

    If you still having issues with your server, please open a ticket.

    We've already had redundancy in place for cases like this, specifically in this case, our Core1 router went down due to power losses from the Datacenter. Leaving Core2 online with our backup carrier.

    However, this backup carrier (Cogent) did not pick up our routes properly, causing our network to go completely down, we will be looking to replace this backup carrier in the coming weeks to prevent this from happening again in the future.

    The cabinet started to go online once the Datacenter fixed the power issue, making our Core1 router back online. Most cabinets were not affected by this power losses, only our core cabinets were affected.

    We apologies for any inconvenience caused.

    Please let us know if you have any further questions on this incident!

  • Update
    Update

    Our team is getting network back online after the power reset affecting our core switches. Routes are being learnt and populated. We will keep you updated.

  • Identified
    Identified

    Coresite has confirmed a power issue in our cabinets and has been working to resolve it.

    Our monitors alerted us that networks are being restored at this time, our network admin team is double checking at this time.

  • Update
    Update

    Our technician is currently en-route to the DC but will be slightly delayed due to heavy traffic in downtown Chicago.


    Our team is currently still waiting for responses from the Datacenter, they have not provided any updates on our case.

  • Update
    Update

    Our team is still waiting for Datacenter (Coresite CH2) Response on this matter. They have acknowledged immediately but we have not received any further information from them yet.

    Our last log indicate that there is power losses in the Datacenter, our team access is completely cut as none of our devices are reachable.

  • Investigating
    Investigating

    Our team is aware of issue that causing unavailability at our CHI1 Datacenter. We are currently investigating this incident.

Oct 2024

NYC network maintenance
  • Completed
    October 20, 2024 at 1:48 PM
    Completed
    October 20, 2024 at 1:48 PM

    The issue has been solved at this time. We no longer see any packet losses or unreachability issues.

    All servers have now been up and remained stable. We've successfully migrated to new Core routers and new network design, which will allow us to scale further without performance penalty.

    Our team will add more capacity next weekend as a part of this upgrade. We will put up another announcement once confirmed.

    Please let us know if you via ticket if you still experience any issues.

    Our apologies for any inconvenience caused during this process.

    Thank you for your patience and understanding!

  • Update
    October 20, 2024 at 11:13 AM
    In progress
    October 20, 2024 at 11:13 AM

    Our team is still working on the issue.

    As noted, majority of servers have been back to normal, we are still observing packet losses toward a few IP ranges and are looking into it.

    Our team will post update once we have more information.

  • Update
    October 20, 2024 at 10:13 AM
    In progress
    October 20, 2024 at 10:13 AM

    Most of our IPs are functional, we are aware of a few subnets still flapping, causing spikes in latency and 5-12% packet losses, our team is investigating on this issue.

    All downlinks to our core routers have been migrated successfully at this time.

  • Update
    October 20, 2024 at 9:13 AM
    In progress
    October 20, 2024 at 9:13 AM

    We are still in the mid of transferring downlinks of old Core router to new, it is expected that servers will experience intermittent packet losses or be unreachable for short period amount of time during this process.

  • Update
    October 20, 2024 at 8:33 AM
    In progress
    October 20, 2024 at 8:33 AM

    Our backup upstream did not pickup our subnets immediately and has resulted in a slight downtime of 5-10 minutes. All subnets have been back online at this time.

  • Update
    October 20, 2024 at 7:56 AM
    In progress
    October 20, 2024 at 7:56 AM

    All of our Edge switches have been updated to latest firmware and configuration at this time. Our team is currently upgrading Core routers to newer hardware. It is expected that your servers will have packet losses or unreachable during this time as we transfer loads back and forth between old and new cores, with new uplinks.

  • In progress
    October 20, 2024 at 6:00 AM
    In progress
    October 20, 2024 at 6:00 AM
    Maintenance is now in progress
  • Planned
    October 20, 2024 at 6:00 AM
    Planned
    October 20, 2024 at 6:00 AM

    We will be performing a network upgrade at our New York location to improve network performance and reliability.


    Scheduled Timeframe: Sunday, October 20, 2024, from 1:00 AM to 12:00 PM (EST).

    During this timeframe, your service may experience periodic packet loss and could become unreachable for up to 15 minutes as we perform software updates across all our switches.

    We have carefully planned this maintenance and do not expect any issues during this process. Our team will promptly post updates as we proceed with the maintenance.

    If you have any questions about this maintenance, please let us know via ticket.

    Thank you for your understanding!

AMS1 Partial Power Outage
  • Resolved
    Resolved

    Both side of powers have been restored at the Datacenter, all Asus servers have been re-installed with redundant PSU at this time.

    We will now close this status as the issue is resolved.

    Please let us know via ticket if you still have any issues!

  • Monitoring
    Monitoring

    We have temporarily detached the failed PSU as a workaround to this issue, the CPU has stopped throttling and all servers are reachable at this time.

    Please note that the servers were not down, just the performance of the servers were heavily degraded.

    Our team has reported this to Asus engineer and they are aware of the issue.

    We will keep monitoring the situation until the Datacenter power fully restored.

    Our apologize for any inconvenience caused!

    Please let us know via ticket if you still having issues.

  • Update
    Update

    Our technician has arrived on-site and is checking on the issue.

  • Identified
    Identified

    We are aware of the issue in our Amsterdam Datacenter (AMS1) causing 4 nodes to be unreachable.

    Our current Datacenter (Equinix AM5) is losing 1 power feed, and all our host nodes have started using redundant power supply, however a few Asus-based host nodes are currently heavily throttled due to missing power supply, it still remains online but the performance has been heavily degraded to the point where it's unresponsive.

    Our technician has been dispatched to the Datacenter to check on the issue. We've also contacted Asus for explanation as to why this behavior occurred.

    Further update will follow as we have more information to share.

    Our apologies for any inconvenience caused.

Oct 2024 to Dec 2024

Next