Hyonix - Notice history

All systems operational

Operational

Hyonix Panel - Operational

Hyonix Backend - Operational

Hyonix API - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Reston, Virginia (IAD1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

Miami, Florida (MIA1) - Operational

London, UK (LON1) - Operational

Amsterdam, NL (AMS1) - Operational

Hong Kong, HK (HKG1) - Operational

Tokyo, JP (TYO1) - Operational

Sydney, AU (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Notice history

Aug 2024

Hyonix Panel Maintenance
  • Completed
    August 03, 2024 at 1:08 PM
    Completed
    August 03, 2024 at 1:08 PM

    Maintenance has now been completed! Both panel and API have been online.

    Please note that your servers are not affected by this maintenance. If you are having issues with your servers, please open a ticket and our team will check on the issues for you!

  • In progress
    August 03, 2024 at 7:45 AM
    In progress
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

  • Planned
    August 03, 2024 at 7:45 AM
    Planned
    August 03, 2024 at 7:45 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

Jul 2024

Hyonix Panel Maintenance.
  • Completed
    July 26, 2024 at 3:22 PM
    Completed
    July 26, 2024 at 3:22 PM

    Maintenance has now been completed! Access to panel and API have been restored.

    Please note that your servers with us are not affected, if you are seeing any issues with the server, kindly open a ticket or send email to hello@hyonix.com

    Thank you for your patience and understanding!

  • Update
    July 26, 2024 at 12:47 PM
    In progress
    July 26, 2024 at 12:47 PM

    The upgrade progress is taking longer than expected. Our team is still actively working on it.

    Please note that all servers remained operational and is not affected by this issue. Support will be delayed until our panel finished upgrading!

    Our apologies for any inconvenience caused.

  • In progress
    July 26, 2024 at 8:11 AM
    In progress
    July 26, 2024 at 8:11 AM

    Maintenance has started.

  • Planned
    July 26, 2024 at 8:10 AM
    Planned
    July 26, 2024 at 8:10 AM

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

LON1 Datacenter Migration
  • Completed
    August 03, 2024 at 1:12 PM
    Completed
    August 03, 2024 at 1:12 PM

    We've completely decommissioned old London Datacenter. This maintenance is now completed as we've been able to confirm that the new London Datacenter is running stable.

    If you have any issues with your London servers, please let us know via ticket.

    Thank you for your patience and understanding on this matter!

  • Update
    July 28, 2024 at 8:26 AM
    In progress
    July 28, 2024 at 8:26 AM

    We are disconnecting old Datacenter at this time. Our team will be closely monitoring the process. It is expected that there will be some packet losses/ brief network disruption during this time as we transfer network loads between 2 Datacenters.

  • Update
    July 27, 2024 at 5:26 PM
    In progress
    July 27, 2024 at 5:26 PM

    We've now completed all host nodes migration, our team is monitoring and re-checking to see if there is any other issues before we completely disconnect the old Datacenter.

    We will post more update if any issues arise.

  • Update
    July 21, 2024 at 8:48 PM
    In progress
    July 21, 2024 at 8:48 PM

    We've completed migration of 90% of our infrastructure. The remaining maintenance will take place next Saturday (London time) to ensure minimal impact to customers.

    Average downtime at this time is 1 hour 34 minutes per node, there were some unexpected issues where hardware were not being recognized or just straight up faulty (after the quality test) after the migration, which required a hard reboot to proceed with replacements.

    Any affected customers will be notified and compensated promptly.

    All nodes are now fully operational and we do not expect any further interruptions until next weekend!

    Thank you for your patience and understanding on this matter.

    If you still seeing issues, please contact us via ticket.

  • Update
    July 20, 2024 at 7:44 PM
    In progress
    July 20, 2024 at 7:44 PM

    We've completed migration of 45% of our infrastructure, we've hit some unexpected issues but all looking good at this time.

    Average downtime thus far is 1 hour 28 minutes per node, still below our estimated downtime!

    Our team will continue migrating the remaining in a few hours - hopefully to push faster progress by tomorrow.

  • Update
    July 20, 2024 at 1:49 PM
    In progress
    July 20, 2024 at 1:49 PM

    We've completed migration of 25% of our infrastructure to the new Datacenter smoothly with very minimal impact.

    Average network downtime is 50 minutes as opposed to the expected 2-4 hours. Affected servers is being announced.

    Our on-site technician is working very slowly to ensure the migration happen smoothly with no errors.

    Please let us know if you have any further questions.

  • In progress
    July 19, 2024 at 9:00 PM
    In progress
    July 19, 2024 at 9:00 PM
    Maintenance is now in progress
  • Planned
    July 19, 2024 at 9:00 PM
    Planned
    July 19, 2024 at 9:00 PM

    Due to the recent quality issues with our current London Datacenter (multiple random overheating issues), we will be performing migration to new Datacenter in the next 2 weeks. Migrations will be done in batches, affected services will be notified approximately 24 hours prior and the migration is expected to happen very seamlessly.

    Our team has prepared interconnection between 2 Datacenters at this time to ensure everything go smoothly during this migration.

    Customers servers will not be shutdown during this process, however, there will be network disruptions as we migrate your data to the new Datacenter. The expected network downtime per server is estimated to be 2-4 hours.

    Our team will only perform migration in off-peak hours to prevent impacts to customer usage, the works will commence at 10PM in Friday, 19th July 2024 UTC+0.

    Please let us know via ticket if you need any further clarification!

Cloudflare DNS not reachable
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Our team is aware of an issue that affect connectivity to Cloudflare from our main upstream (Global Secure Layer).

    The root cause is that BHARTI Airtel (Cloudflare's upstream) mistakenly leaked Cloudflare prefixes in Los Angeles (while it should have stayed in India), causing network disruption for all networks that have PNI (Private network interconnection) with them. One of our upstream has PNI with them were impacted (due to having global backbone - all sites are interconnected and use Airtel's invalid route to Cloudflare instead).

    Most of the servers deployed on our platform using Cloudflare DNS as default, which is why the DNS error occurred. This issue is limited to Cloudflare only, all other networks were not impacted.

    The incident has lasted for 8 minutes and seems to be recovered at this time as Airtel has quickly fixed the leakage.

    This is not limited to us but is a widespread issue with other providers as well: https://downdetector.com/status/cloudflare/

    If you still experiencing the issue, please let us know via ticket!

    Thank you for your understanding.

LON1 Network becoming unstable
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We were able to confirm that the issue was solved after multiple tests from multiple sources.

    Our team will schedule a core router upgrade in the coming days to fix this issue permanently. For now, no further impact is anticipated.

    We sincerely apologize for any inconvenience caused.

    Please let us know if you still experiencing any issues!

  • Update
    Update

    We've completed the rollback and everything looks to be restoring to normal. We were still monitoring the issue closely.

  • Identified
    Identified

    After extensive troubleshooting effort and gathering data from other sources (upstream, partners), we've been able to narrow down the issue to one of our Cores not forwarding packets properly, causing an internal ARP flood. This issue was caused due to an Arista software bug after we converted our network to VXLAN-EVPN based.

    Our team began to rollback all the changes.

  • Investigating
    Investigating

    This is a snapshot of the incident timeframe, status was posted after the incident concluded due to lack of human resources.

    We were aware of the issue where LON1 would randomly dropping packets. Our team is currently waiting for responses from all related parties.

Jun 2024

NYC1 Outbound network not working
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Our alt upstream has fixed the issue and we've put this upstream to backup with multiple prepends at this time.

    Our team will monitor the issue for another 72 hours before closing this status.

    Thank you for your patience and understanding!

  • Identified
    Identified

    We are aware of an issue that affect outgoing route in our NYC1 location. Our team was able to identify the issue started with an upstream and has now disabled this uplink.

    The issue only affect customers using Cloudflare services (DNS).

    Everything has been back normal at this time, we are in exchange with the affected upstream and will post update once we have more information.

Jun 2024 to Aug 2024

Next