Hey there! We've found the following might help you get unblocked faster:

• 🧵 DNSPROBEFINISHED_NXDOMAIN error

• 🧵 Migration to Railway Metal with little downtime

• 🧵 Urgent: Database Serverless Not Waking — Suspected After Metal Upgrade (Affecting Customers)

If you find the answer from one of these, please let us know by solving the thread!

Hello there James,

Unfortunately yes, that workload was on a host that had to be restarted. It appears when calling the incident, the internal tool I use is referencing the old host set. I apologize and I will rectify this moving forward. Hopefully this didn't lead to much impact.

Hi Angelo, so to confirm, the status page incident that James linked was the underlying issue that caused our service to become unavailable, and the incident was tagged with the incorrect metadata about what cluster was impacted (since the incident says GCP, but the impacted service was on Metal)?

Will there be more flavor shared about the nature of the underlying issue? The timestamps on the incident linked don't appear to align with what we observed; it would be great to learn more what went wrong.

Yes for the above.

I am unsure what you mean by the timestamps, at that timeframe we did call when we noticed that single host went unhealthy. It may be delayed but it does line up on our side. As for more information, I am happy to, Railway continually updates our hosts throughout the day and during a rollout the update daemon locked the machine. It was a combination of multiple services that were "hot" such as logging and other services that were at load that didn't gracefully shutdown locking the machine. We then rebooted the machine that restored access to the workload.

We did recover the machine but the Infra team, is working through a core fix and in the meantime isolated that box from any new workloads so that we can make this hosthang re-producable.

That said, and I never like to hold customer workloads hostage, but your current plan has no SLA, so it's currently at that service level. If this is a critical service that can't be impacted at all, we can see about getting you on a dedicated host.

By timestamps I mean that we observed impact starting about 15 minutes before the recorded start of impact on the status page.

Yes, that is likely, we do need time to react to pages and confirm if it is an incident.