Hyonix - Notice history

All systems operational

Operational

Hyonix Panel - Operational

Hyonix Backend - Operational

Hyonix API - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Notice history

Sep 2024

NYC2 Outage
  • Resolved
    Resolved

    This incident has been marked as resolved as there has been no further issues after 7 days!

  • Monitoring
    Monitoring

    The Datacenter Operators have confirmed that maintenance has been completed, there is no further works being done, all powers have been restored.

    Our team will still monitor the situation for 1-2 days before closing this incident.

    We apologizes for any issues or inconvenience may have caused.

    Thank you for your patience and understanding during this process.

    Please let us know via ticket if you still have any issues!

  • Update
    Update

    We've completed assessment on this incident.


    Root cause: The Datacenter is performing maintenance related to electrical system upgrade. During the maintenance window, the power circuits of the cabinet row hosting our main network cabinet has tripped, causing our routers to lose both main and redundant power sources, leading to all other cabinets losing network access.


    Solution: Upon arrival, our technician noticed that many electricians were aware of the problem and were already actively working on it. The power was restored shortly after, however due to the power getting completely tripped, a few of our cabinets were affected. Our tech team started to go over every cabinet to ensure that all servers were brought online.


    Current state: All servers are back online and operational at this time. The Datacenter is still under maintenance but don't expect any further issues. We were given ETA for completion to be at 09/18/2024 04:00:00 am EDT. Our team will update once we heard confirmation from the Datacenter.

    Customers with servers restarted due to the power loss will be notified via ticket. Our team will be monitoring the issue closely.

    Our apologies for any inconvenience caused.

    Please reach out to us via ticket if you still having issues with your servers.

  • Update
    Update

    Datacenter Operators confirmed the power outage in a section of the Datacenter hall. Affecting many rows of cabinets. Electricians are also on site and bringing the power back up.

    Our team is currently assessing the impact of the power outages and will post further update shortly.

  • Identified
    Identified

    Our technician is currently on-site at this time, it appears the Datacenter is suffering a major power outages. we are still checking with Datacenter Operators on the root cause.

    More update to follow!

  • Investigating
    Investigating

    Our team is aware of an issue related to our NYC2 Datacenter being down. NYC1 still remains fully operational.

    The issue started at: 02:27 AM UTC. We've immediately dispatched our technician to the Datacenter to check on the status.

    We are looking into this issue at this time!

    Our team will post updates as we receive news from the Datacenter.

Aug 2024

Hyonix Panel Maintenance
  • Completed
    August 03, 2024 at 1:08 PM
    Completed
    August 03, 2024 at 1:08 PM

    Maintenance has now been completed! Both panel and API have been online.

    Please note that your servers are not affected by this maintenance. If you are having issues with your servers, please open a ticket and our team will check on the issues for you!

  • In progress
    August 03, 2024 at 7:45 AM
    In progress
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

  • Planned
    August 03, 2024 at 7:45 AM
    Planned
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

Jul 2024

Hyonix Panel Maintenance.
  • Completed
    July 26, 2024 at 3:22 PM
    Completed
    July 26, 2024 at 3:22 PM

    Maintenance has now been completed! Access to panel and API have been restored.

    Please note that your servers with us are not affected, if you are seeing any issues with the server, kindly open a ticket or send email to hello@hyonix.com

    Thank you for your patience and understanding!

  • Update
    July 26, 2024 at 12:47 PM
    In progress
    July 26, 2024 at 12:47 PM

    The upgrade progress is taking longer than expected. Our team is still actively working on it.

    Please note that all servers remained operational and is not affected by this issue. Support will be delayed until our panel finished upgrading!

    Our apologies for any inconvenience caused.

  • In progress
    July 26, 2024 at 8:11 AM
    In progress
    July 26, 2024 at 8:11 AM

    Maintenance has started.

  • Planned
    July 26, 2024 at 8:10 AM
    Planned
    July 26, 2024 at 8:10 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

LON1 Datacenter Migration
  • Completed
    August 03, 2024 at 1:12 PM
    Completed
    August 03, 2024 at 1:12 PM

    We've completely decommissioned old London Datacenter. This maintenance is now completed as we've been able to confirm that the new London Datacenter is running stable.

    If you have any issues with your London servers, please let us know via ticket.

    Thank you for your patience and understanding on this matter!

  • Update
    July 28, 2024 at 8:26 AM
    In progress
    July 28, 2024 at 8:26 AM

    We are disconnecting old Datacenter at this time. Our team will be closely monitoring the process. It is expected that there will be some packet losses/ brief network disruption during this time as we transfer network loads between 2 Datacenters.

  • Update
    July 27, 2024 at 5:26 PM
    In progress
    July 27, 2024 at 5:26 PM

    We've now completed all host nodes migration, our team is monitoring and re-checking to see if there is any other issues before we completely disconnect the old Datacenter.

    We will post more update if any issues arise.

  • Update
    July 21, 2024 at 8:48 PM
    In progress
    July 21, 2024 at 8:48 PM

    We've completed migration of 90% of our infrastructure. The remaining maintenance will take place next Saturday (London time) to ensure minimal impact to customers.

    Average downtime at this time is 1 hour 34 minutes per node, there were some unexpected issues where hardware were not being recognized or just straight up faulty (after the quality test) after the migration, which required a hard reboot to proceed with replacements.

    Any affected customers will be notified and compensated promptly.

    All nodes are now fully operational and we do not expect any further interruptions until next weekend!

    Thank you for your patience and understanding on this matter.

    If you still seeing issues, please contact us via ticket.

  • Update
    July 20, 2024 at 7:44 PM
    In progress
    July 20, 2024 at 7:44 PM

    We've completed migration of 45% of our infrastructure, we've hit some unexpected issues but all looking good at this time.

    Average downtime thus far is 1 hour 28 minutes per node, still below our estimated downtime!

    Our team will continue migrating the remaining in a few hours - hopefully to push faster progress by tomorrow.

  • Update
    July 20, 2024 at 1:49 PM
    In progress
    July 20, 2024 at 1:49 PM

    We've completed migration of 25% of our infrastructure to the new Datacenter smoothly with very minimal impact.

    Average network downtime is 50 minutes as opposed to the expected 2-4 hours. Affected servers is being announced.

    Our on-site technician is working very slowly to ensure the migration happen smoothly with no errors.

    Please let us know if you have any further questions.

  • In progress
    July 19, 2024 at 9:00 PM
    In progress
    July 19, 2024 at 9:00 PM
    Maintenance is now in progress
  • Planned
    July 19, 2024 at 9:00 PM
    Planned
    July 19, 2024 at 9:00 PM

    Due to the recent quality issues with our current London Datacenter (multiple random overheating issues), we will be performing migration to new Datacenter in the next 2 weeks. Migrations will be done in batches, affected services will be notified approximately 24 hours prior and the migration is expected to happen very seamlessly.

    Our team has prepared interconnection between 2 Datacenters at this time to ensure everything go smoothly during this migration.

    Customers servers will not be shutdown during this process, however, there will be network disruptions as we migrate your data to the new Datacenter. The expected network downtime per server is estimated to be 2-4 hours.

    Our team will only perform migration in off-peak hours to prevent impacts to customer usage, the works will commence at 10PM in Friday, 19th July 2024 UTC+0.

    Please let us know via ticket if you need any further clarification!

Cloudflare DNS not reachable
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Our team is aware of an issue that affect connectivity to Cloudflare from our main upstream (Global Secure Layer).

    The root cause is that BHARTI Airtel (Cloudflare's upstream) mistakenly leaked Cloudflare prefixes in Los Angeles (while it should have stayed in India), causing network disruption for all networks that have PNI (Private network interconnection) with them. One of our upstream has PNI with them were impacted (due to having global backbone - all sites are interconnected and use Airtel's invalid route to Cloudflare instead).

    Most of the servers deployed on our platform using Cloudflare DNS as default, which is why the DNS error occurred. This issue is limited to Cloudflare only, all other networks were not impacted.

    The incident has lasted for 8 minutes and seems to be recovered at this time as Airtel has quickly fixed the leakage.

    This is not limited to us but is a widespread issue with other providers as well: https://downdetector.com/status/cloudflare/

    If you still experiencing the issue, please let us know via ticket!

    Thank you for your understanding.

LON1 Network becoming unstable
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We were able to confirm that the issue was solved after multiple tests from multiple sources.

    Our team will schedule a core router upgrade in the coming days to fix this issue permanently. For now, no further impact is anticipated.

    We sincerely apologize for any inconvenience caused.

    Please let us know if you still experiencing any issues!

  • Update
    Update

    We've completed the rollback and everything looks to be restoring to normal. We were still monitoring the issue closely.

  • Identified
    Identified

    After extensive troubleshooting effort and gathering data from other sources (upstream, partners), we've been able to narrow down the issue to one of our Cores not forwarding packets properly, causing an internal ARP flood. This issue was caused due to an Arista software bug after we converted our network to VXLAN-EVPN based.

    Our team began to rollback all the changes.

  • Investigating
    Investigating

    This is a snapshot of the incident timeframe, status was posted after the incident concluded due to lack of human resources.

    We were aware of the issue where LON1 would randomly dropping packets. Our team is currently waiting for responses from all related parties.

Jul 2024 to Sep 2024

Next