Hyonix - Notice history

All systems operational

Services
Operational

Hyonix Panel - Operational

Hyonix Backend - Operational

Hyonix API - Operational

Virtualization Hosts
Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Core Networks
Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Notice history

Oct 2024

NYC network maintenance
  • Completed
    October 20, 2024 at 1:48 PM
    Completed
    October 20, 2024 at 1:48 PM

    The issue has been solved at this time. We no longer see any packet losses or unreachability issues.

    All servers have now been up and remained stable. We've successfully migrated to new Core routers and new network design, which will allow us to scale further without performance penalty.

    Our team will add more capacity next weekend as a part of this upgrade. We will put up another announcement once confirmed.

    Please let us know if you via ticket if you still experience any issues.

    Our apologies for any inconvenience caused during this process.

    Thank you for your patience and understanding!

  • Update
    October 20, 2024 at 11:13 AM
    In progress
    October 20, 2024 at 11:13 AM

    Our team is still working on the issue.

    As noted, majority of servers have been back to normal, we are still observing packet losses toward a few IP ranges and are looking into it.

    Our team will post update once we have more information.

  • Update
    October 20, 2024 at 10:13 AM
    In progress
    October 20, 2024 at 10:13 AM

    Most of our IPs are functional, we are aware of a few subnets still flapping, causing spikes in latency and 5-12% packet losses, our team is investigating on this issue.

    All downlinks to our core routers have been migrated successfully at this time.

  • Update
    October 20, 2024 at 9:13 AM
    In progress
    October 20, 2024 at 9:13 AM

    We are still in the mid of transferring downlinks of old Core router to new, it is expected that servers will experience intermittent packet losses or be unreachable for short period amount of time during this process.

  • Update
    October 20, 2024 at 8:33 AM
    In progress
    October 20, 2024 at 8:33 AM

    Our backup upstream did not pickup our subnets immediately and has resulted in a slight downtime of 5-10 minutes. All subnets have been back online at this time.

  • Update
    October 20, 2024 at 7:56 AM
    In progress
    October 20, 2024 at 7:56 AM

    All of our Edge switches have been updated to latest firmware and configuration at this time. Our team is currently upgrading Core routers to newer hardware. It is expected that your servers will have packet losses or unreachable during this time as we transfer loads back and forth between old and new cores, with new uplinks.

  • In progress
    October 20, 2024 at 6:00 AM
    In progress
    October 20, 2024 at 6:00 AM
    Maintenance is now in progress
  • Planned
    October 20, 2024 at 6:00 AM
    Planned
    October 20, 2024 at 6:00 AM

    We will be performing a network upgrade at our New York location to improve network performance and reliability.


    Scheduled Timeframe: Sunday, October 20, 2024, from 1:00 AM to 12:00 PM (EST).

    During this timeframe, your service may experience periodic packet loss and could become unreachable for up to 15 minutes as we perform software updates across all our switches.

    We have carefully planned this maintenance and do not expect any issues during this process. Our team will promptly post updates as we proceed with the maintenance.

    If you have any questions about this maintenance, please let us know via ticket.

    Thank you for your understanding!

AMS1 Partial Power Outage
  • Resolved
    Resolved

    Both side of powers have been restored at the Datacenter, all Asus servers have been re-installed with redundant PSU at this time.

    We will now close this status as the issue is resolved.

    Please let us know via ticket if you still have any issues!

  • Monitoring
    Monitoring

    We have temporarily detached the failed PSU as a workaround to this issue, the CPU has stopped throttling and all servers are reachable at this time.

    Please note that the servers were not down, just the performance of the servers were heavily degraded.

    Our team has reported this to Asus engineer and they are aware of the issue.

    We will keep monitoring the situation until the Datacenter power fully restored.

    Our apologize for any inconvenience caused!

    Please let us know via ticket if you still having issues.

  • Identified
    Update

    Our technician has arrived on-site and is checking on the issue.

  • Identified
    Identified

    We are aware of the issue in our Amsterdam Datacenter (AMS1) causing 4 nodes to be unreachable.

    Our current Datacenter (Equinix AM5) is losing 1 power feed, and all our host nodes have started using redundant power supply, however a few Asus-based host nodes are currently heavily throttled due to missing power supply, it still remains online but the performance has been heavily degraded to the point where it's unresponsive.

    Our technician has been dispatched to the Datacenter to check on the issue. We've also contacted Asus for explanation as to why this behavior occurred.

    Further update will follow as we have more information to share.

    Our apologies for any inconvenience caused.

Sep 2024

NYC2 Outage
  • Resolved
    Resolved

    This incident has been marked as resolved as there has been no further issues after 7 days!

  • Monitoring
    Monitoring

    The Datacenter Operators have confirmed that maintenance has been completed, there is no further works being done, all powers have been restored.

    Our team will still monitor the situation for 1-2 days before closing this incident.

    We apologizes for any issues or inconvenience may have caused.

    Thank you for your patience and understanding during this process.

    Please let us know via ticket if you still have any issues!

  • Identified
    Update

    We've completed assessment on this incident.


    Root cause: The Datacenter is performing maintenance related to electrical system upgrade. During the maintenance window, the power circuits of the cabinet row hosting our main network cabinet has tripped, causing our routers to lose both main and redundant power sources, leading to all other cabinets losing network access.


    Solution: Upon arrival, our technician noticed that many electricians were aware of the problem and were already actively working on it. The power was restored shortly after, however due to the power getting completely tripped, a few of our cabinets were affected. Our tech team started to go over every cabinet to ensure that all servers were brought online.


    Current state: All servers are back online and operational at this time. The Datacenter is still under maintenance but don't expect any further issues. We were given ETA for completion to be at 09/18/2024 04:00:00 am EDT. Our team will update once we heard confirmation from the Datacenter.

    Customers with servers restarted due to the power loss will be notified via ticket. Our team will be monitoring the issue closely.

    Our apologies for any inconvenience caused.

    Please reach out to us via ticket if you still having issues with your servers.

  • Identified
    Update

    Datacenter Operators confirmed the power outage in a section of the Datacenter hall. Affecting many rows of cabinets. Electricians are also on site and bringing the power back up.

    Our team is currently assessing the impact of the power outages and will post further update shortly.

  • Identified
    Identified

    Our technician is currently on-site at this time, it appears the Datacenter is suffering a major power outages. we are still checking with Datacenter Operators on the root cause.

    More update to follow!

  • Investigating
    Investigating

    Our team is aware of an issue related to our NYC2 Datacenter being down. NYC1 still remains fully operational.

    The issue started at: 02:27 AM UTC. We've immediately dispatched our technician to the Datacenter to check on the status.

    We are looking into this issue at this time!

    Our team will post updates as we receive news from the Datacenter.

Aug 2024

Hyonix Panel Maintenance
  • Completed
    August 03, 2024 at 1:08 PM
    Completed
    August 03, 2024 at 1:08 PM

    Maintenance has now been completed! Both panel and API have been online.

    Please note that your servers are not affected by this maintenance. If you are having issues with your servers, please open a ticket and our team will check on the issues for you!

  • In progress
    August 03, 2024 at 7:45 AM
    In progress
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

  • Planned
    August 03, 2024 at 7:45 AM
    Planned
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

Aug 2024 to Oct 2024

Next