19 days ago
Once on Friday and twice on Saturday I received emails saying:
A server running your service has experienced a hardware failure. We are actively working to restore it. No action is needed on your part — your service will automatically resume once the issue is resolved.
I already reviewed this thread that had a similar question. What I wanted to ask was why is this happening repeatedly? A hardware failure on a server is something I could understand happens once in a while but three times over this past weekend is worrisome. There's not even a status page that's published where I can see these events and how long and often they occur.
Did I just run into bad luck with this occurring three times in two days? Thank you.
3 Replies
19 days ago
We can confirm three separate host hardware failure events affected your Postgres service, starting at 02:09 UTC, 15:52 UTC and 17:38 UTC on October 31. All three are resolved, and each time the service was moved to a healthy server and restarted. These single-server events are sent as direct notifications to affected customers rather than posted on the public status page.
Our records of these events do not establish why your service was hit three times in that window, so we can't say whether it was bad luck. Each failure takes down a single server, and your database runs on a single server, so for mission-critical workloads we advise using a high availability setup, such as the primary-replica Postgres HA template, so one failed server no longer takes the database offline.
Status changed to Awaiting User Response Railway • 19 days ago
19 days ago
How wide spread have these server outages been? What percent of customers were affected over the weekend?
Status changed to Awaiting Railway Response Railway • 19 days ago
19 days ago
We don't have a figure for what share of customers were affected over the weekend. What we can see is limited to your account: the three notifications were each a failure of a single server your Postgres service was running on. Each one is resolved, and each time the service was moved to a healthy server.
Status changed to Awaiting User Response Railway • 19 days ago
12 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 12 days ago