Hyonix - Notice history

All systems operational

Operational

Hyonix Panel - Operational

Hyonix Backend - Operational

Hyonix API - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC2) - Operational

Miami, Florida (MIA1) - Operational

London, United Kingdom (LON1) - Operational

Amsterdam, Netherlands (AMS2) - Operational

Hong Kong, Hong Kong (HKG1) - Operational

Tokyo, Japan (TYO1) - Operational

Sydney, Australia (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Singapore, Singapore (SIN1) - Operational

Operational

Los Angeles, California (LAX1) - Operational

Dallas, Texas (DAL1) - Operational

Ashburn, Virginia (IAD2) - Operational

New York, New York (NYC1) - Operational

New York, New York (NYC2) - Operational

Miami, Florida (MIA1) - Operational

London, United Kingdom (LON1) - Operational

London, United Kingdom (LON-THN) - Operational

Slough, United Kingdom (LON5) - Operational

Amsterdam, Netherlands (AMS2) - Operational

Amsterdam, Netherlands (AMS-NIKHEF) - Operational

Hong Kong, Hong Kong (HKG1) - Operational

Tokyo, Japan (TYO1) - Operational

Sydney, Australia (SYD1) - Operational

Frankfurt, Germany (FRA10) - Operational

Toronto, Canada (TOR10) - Operational

Chicago, Illinois (CHI1) - Operational

Singapore, Singapore (SIN1) - Operational

Notice history

View current status

Aug 2024

AUG3
Hyonix Panel Maintenance
CompletedMaintenance5 hours 23 minutes
  • CompletedAugust 03, 2024 at 1:08 PMUTC

    Maintenance has now been completed! Both panel and API have been online.

    Please note that your servers are not affected by this maintenance. If you are having issues with your servers, please open a ticket and our team will check on the issues for you!

  • PlannedAugust 03, 2024 at 7:45 AMUTC

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

  • In progressAugust 03, 2024 at 7:45 AMUTC

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

Jul 2024

JUL26
Hyonix Panel Maintenance.
CompletedMaintenance7 hours 11 minutes
  • CompletedJuly 26, 2024 at 3:22 PMUTC

    Maintenance has now been completed! Access to panel and API have been restored.

    Please note that your servers with us are not affected, if you are seeing any issues with the server, kindly open a ticket or send email to hello@hyonix.com

    Thank you for your patience and understanding!

  • UpdateJuly 26, 2024 at 12:47 PMUTC

    The upgrade progress is taking longer than expected. Our team is still actively working on it.

    Please note that all servers remained operational and is not affected by this issue. Support will be delayed until our panel finished upgrading!

    Our apologies for any inconvenience caused.

  • In progressJuly 26, 2024 at 8:11 AMUTC

    Maintenance has started.

  • PlannedJuly 26, 2024 at 8:10 AMUTC

    We are performing an upgrade on our panel at the moment. It's estimated to be completed by 1 PM UTC+0.

    If you have any issues, please let us know via email: hello@hyonix.com.

    Our apologies for any inconvenience caused.

    Thank you for your patience and understanding!

JUL19
LON1 Datacenter Migration
CompletedMaintenance352 hours 12 minutes
  • CompletedAugust 03, 2024 at 1:12 PMUTC

    We've completely decommissioned old London Datacenter. This maintenance is now completed as we've been able to confirm that the new London Datacenter is running stable.

    If you have any issues with your London servers, please let us know via ticket.

    Thank you for your patience and understanding on this matter!

  • UpdateJuly 28, 2024 at 8:26 AMUTC

    We are disconnecting old Datacenter at this time. Our team will be closely monitoring the process. It is expected that there will be some packet losses/ brief network disruption during this time as we transfer network loads between 2 Datacenters.

  • UpdateJuly 27, 2024 at 5:26 PMUTC

    We've now completed all host nodes migration, our team is monitoring and re-checking to see if there is any other issues before we completely disconnect the old Datacenter.

    We will post more update if any issues arise.

  • UpdateJuly 21, 2024 at 8:48 PMUTC

    We've completed migration of 90% of our infrastructure. The remaining maintenance will take place next Saturday (London time) to ensure minimal impact to customers.

    Average downtime at this time is 1 hour 34 minutes per node, there were some unexpected issues where hardware were not being recognized or just straight up faulty (after the quality test) after the migration, which required a hard reboot to proceed with replacements.

    Any affected customers will be notified and compensated promptly.

    All nodes are now fully operational and we do not expect any further interruptions until next weekend!

    Thank you for your patience and understanding on this matter.

    If you still seeing issues, please contact us via ticket.

  • UpdateJuly 20, 2024 at 7:44 PMUTC

    We've completed migration of 45% of our infrastructure, we've hit some unexpected issues but all looking good at this time.

    Average downtime thus far is 1 hour 28 minutes per node, still below our estimated downtime!

    Our team will continue migrating the remaining in a few hours - hopefully to push faster progress by tomorrow.

  • UpdateJuly 20, 2024 at 1:49 PMUTC

    We've completed migration of 25% of our infrastructure to the new Datacenter smoothly with very minimal impact.

    Average network downtime is 50 minutes as opposed to the expected 2-4 hours. Affected servers is being announced.

    Our on-site technician is working very slowly to ensure the migration happen smoothly with no errors.

    Please let us know if you have any further questions.

  • In progressJuly 19, 2024 at 9:00 PMUTC
    Maintenance is now in progress
  • PlannedJuly 19, 2024 at 9:00 PMUTC

    Due to the recent quality issues with our current London Datacenter (multiple random overheating issues), we will be performing migration to new Datacenter in the next 2 weeks. Migrations will be done in batches, affected services will be notified approximately 24 hours prior and the migration is expected to happen very seamlessly.

    Our team has prepared interconnection between 2 Datacenters at this time to ensure everything go smoothly during this migration.

    Customers servers will not be shutdown during this process, however, there will be network disruptions as we migrate your data to the new Datacenter. The expected network downtime per server is estimated to be 2-4 hours.

    Our team will only perform migration in off-peak hours to prevent impacts to customer usage, the works will commence at 10PM in Friday, 19th July 2024 UTC+0.

    Please let us know via ticket if you need any further clarification!

Cloudflare DNS not reachable
Resolved138 hours 35 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Our team is aware of an issue that affect connectivity to Cloudflare from our main upstream (Global Secure Layer).

    The root cause is that BHARTI Airtel (Cloudflare's upstream) mistakenly leaked Cloudflare prefixes in Los Angeles (while it should have stayed in India), causing network disruption for all networks that have PNI (Private network interconnection) with them. One of our upstream has PNI with them were impacted (due to having global backbone - all sites are interconnected and use Airtel's invalid route to Cloudflare instead).

    Most of the servers deployed on our platform using Cloudflare DNS as default, which is why the DNS error occurred. This issue is limited to Cloudflare only, all other networks were not impacted.

    The incident has lasted for 8 minutes and seems to be recovered at this time as Airtel has quickly fixed the leakage.

    This is not limited to us but is a widespread issue with other providers as well: https://downdetector.com/status/cloudflare/

    If you still experiencing the issue, please let us know via ticket!

    Thank you for your understanding.

LON1 Network becoming unstable
ResolvedPartial outage210 hours 54 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    We were able to confirm that the issue was solved after multiple tests from multiple sources.

    Our team will schedule a core router upgrade in the coming days to fix this issue permanently. For now, no further impact is anticipated.

    We sincerely apologize for any inconvenience caused.

    Please let us know if you still experiencing any issues!

  • UpdateUTC

    We've completed the rollback and everything looks to be restoring to normal. We were still monitoring the issue closely.

  • IdentifiedUTC

    After extensive troubleshooting effort and gathering data from other sources (upstream, partners), we've been able to narrow down the issue to one of our Cores not forwarding packets properly, causing an internal ARP flood. This issue was caused due to an Arista software bug after we converted our network to VXLAN-EVPN based.

    Our team began to rollback all the changes.

  • InvestigatingUTC

    This is a snapshot of the incident timeframe, status was posted after the incident concluded due to lack of human resources.

    We were aware of the issue where LON1 would randomly dropping packets. Our team is currently waiting for responses from all related parties.

Jun 2024

NYC1 Outbound network not working
ResolvedDegraded performance615 hours 3 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    Our alt upstream has fixed the issue and we've put this upstream to backup with multiple prepends at this time.

    Our team will monitor the issue for another 72 hours before closing this status.

    Thank you for your patience and understanding!

  • IdentifiedUTC

    We are aware of an issue that affect outgoing route in our NYC1 location. Our team was able to identify the issue started with an upstream and has now disabled this uplink.

    The issue only affect customers using Cloudflare services (DNS).

    Everything has been back normal at this time, we are in exchange with the affected upstream and will post update once we have more information.

Previous

Jun 2024 to Aug 2024

Next