Hi, I got a "Server Outage Affecting Your Service" email at ~5:57 AM today about a hardware failure on my production Postgres, with an estimated 30-minute resolution. It's now 4+ hours later and the database is still unreachable. Railway dashboard (Postgres service, Database tab) shows: - Deployment Online: OK - Required Variables: OK - Database Connection: FAILED — "unable to connect to the database via SSH. Connection terminated unexpectedly." So the service reads as "Online," but the Postgres server isn't accepting connections. What I'm seeing externally, consistent with that: - The TCP proxy accepts a connection, but queries never complete. A plain SELECT 1 hangs and times out (~20s) every time, for hours now. - My app service is healthy, but anything that touches the DB hangs until timeout. Questions: 1. What's the current recovery status and a realistic ETA? "Connection terminated unexpectedly" suggests the instance or its volume didn't fully recover from the hardware event. 2. Can you confirm there's no data loss, and if there's any risk, what the restore point is? 3. If the instance can't be recovered quickly, what's the fastest safe path to restore from the latest backup to a healthy volume? I'd rather restore now than keep waiting if recovery is stalled. My account is logged in here, so you should be able to pull the specific project and service. Happy to share anything else you need privately. This is a production database and it's well past the estimate, so I'd appreciate a priority look. Thanks, Rudy

Production Postgres "Online" but unreachable 4+ hrs after hardware-failure notice — "Connection terminated unexpectedly"

rudymdc

HOBBYOP

a month ago

Hi,

I got a "Server Outage Affecting Your Service" email at ~5:57 AM today about a hardware failure on my production Postgres, with an estimated 30-minute resolution. It's now 4+ hours later and the database is still unreachable.

Railway dashboard (Postgres service, Database tab) shows:

Deployment Online: OK
Required Variables: OK
Database Connection: FAILED — "unable to connect to the database via SSH. Connection terminated unexpectedly."

So the service reads as "Online," but the Postgres server isn't accepting connections.

What I'm seeing externally, consistent with that:

The TCP proxy accepts a connection, but queries never complete. A plain SELECT 1 hangs and times out (~20s) every time, for hours now.
My app service is healthy, but anything that touches the DB hangs until timeout.

Questions:

What's the current recovery status and a realistic ETA? "Connection terminated unexpectedly" suggests the instance or its volume didn't fully recover from the hardware event.
Can you confirm there's no data loss, and if there's any risk, what the restore point is?
If the instance can't be recovered quickly, what's the fastest safe path to restore from the latest backup to a healthy volume? I'd rather restore now than keep waiting if recovery is stalled.

My account is logged in here, so you should be able to pull the specific project and service. Happy to share anything else you need privately. This is a production database and it's well past the estimate, so I'd appreciate a priority look.

Thanks,

Rudy

Solved

2 Replies

Status changed to Awaiting Railway Response Railway • about 1 month ago

rudymdc

HOBBYOP

a month ago

Adding the precise timeline and error signature from my application logs, in case it helps pinpoint the instance/volume.

Times below are UTC (my outage email was 5:57 AM CST, which is 10:57 UTC, so it lines up with the failures here):

Until ~10:38 UTC: database healthy and serving normally (successful queries, normal Postgres checkpoints in the logs).
10:54 UTC: first failure, asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation (after the request hung ~900s).
10:57 UTC: I received the hardware-failure outage notice (30 minute estimate).
10:59 UTC: my logs show a volume remount event (Mounting volume on: ...), consistent with the instance moving after the hardware failure.
Since then: every new connection fails at the driver level. asyncpg cannot establish a connection and times out (asyncio TimeoutError in create_connection), so all DB-backed requests 500 after ~60s. Most recent failure 16:07 UTC.

So the pattern is: it was fine, the volume remounted around the hardware event, and Postgres has not accepted a new connection since. Still down as I write this, now 5+ hours.

Same three asks as above: recovery status/ETA, confirmation of no data loss (and restore point if there's any risk), and if recovery is stalled, the fastest safe path to restore from my latest backup to a healthy volume.

sam-a

EMPLOYEE

a month ago

Your volume is intact with data on it (about 122 MB, state READY), so there is no data loss. The Postgres process itself needs to be restarted cleanly. Open your Postgres service, press Cmd+K (or Ctrl+K) to open the command palette, and select "Redeploy source image" - this re-pulls a fresh image and should restore connectivity. A normal redeploy from the three-dot menu may not be enough here, so use the command palette method. If the database is still unreachable after that, reply here and we can dig further.

Status changed to Awaiting User Response Railway • about 1 month ago

Status changed to Solved rudymdc • about 1 month ago

Welcome!