18 days ago
Hi Railway team!
On June 18 at 11:41 I received an outage notice about a hardware failure affecting my Postgres service (project modest-friendship, environment production). The notice gave an expected resolution of 30 minutes. It is now over 25 hours later and the database is still down.
Current state: the Postgres service shows "Online" in the dashboard, but the database connection check fails with "Connection terminated unexpectedly" and is not accepting connections. My application (companion_os, getcompanionos.com) has been crash-looping against the unreachable database since the failure, so the site has been offline for over a day.
Could you please:
- Restore the Postgres database, or give me a realistic ETA.
- Confirm whether my data and the postgres-volume are intact, and whether a backup is available in case the volume was affected.
This is a live production service. Thank you for the help.
I write it here, because this is what your email said I should do.
7 Replies
18 days ago
Note: I have no backups configured, so confirmation that the postgres-volume survived intact is critical for me.
0x5b62656e5d
Try redeploying your database.
18 days ago
Thanks for the quick reply.
To clarify: I've already redeployed my application service (companion_os). I have not redeployed the Postgres service itself yet, do you mean redeploying Postgres specifically?
Before I do, I want to be sure I don't make it worse. Since this started as a hardware failure on the server running Postgres, and I have no backups configured, could you confirm: will redeploying the Postgres service remount the existing volume with my data intact, or is there any risk it provisions a fresh/empty volume or loses data if the underlying volume was affected by the failure?
If you can confirm it keeps the existing data, I'll redeploy right away. Thank you.
0x5b62656e5d
Also, which region is this database deployed in?
18 days ago
EU West
liinasuoniemi
Thanks for the quick reply. To clarify: I've already redeployed my application service (companion_os). I have not redeployed the Postgres service itself yet, do you mean redeploying Postgres specifically? Before I do, I want to be sure I don't make it worse. Since this started as a hardware failure on the server running Postgres, and I have no backups configured, could you confirm: will redeploying the Postgres service remount the existing volume with my data intact, or is there any risk it provisions a fresh/empty volume or loses data if the underlying volume was affected by the failure? If you can confirm it keeps the existing data, I'll redeploy right away. Thank you.
18 days ago
The volume itself should be intact. Also, I'd just leave it alone for now until the issue is resolved.
(Related thread: https://station.railway.com/questions/ostgres-service-online-but-private-conn-94aaa964)
0x5b62656e5d
The volume itself should be intact. Also, I'd just leave it alone for now until the issue is resolved. (Related thread: https://station.railway.com/questions/ostgres-service-online-but-private-conn-94aaa964)
18 days ago
Thank you, that's reassuring on the volume. I'll leave it alone and wait for the incident to resolve. Could you update this thread once it's fixed? And thanks for the related link, good to know it's being worked on. Appreciate the help.