Skip to content
Production Postgres stuck on deploy after today's hardware failure (Oct 31)
felipevt
OP

19 days ago

Hi Railway team,

Since the hardware failure today (Oct 31, around 16:00 UTC), our production Postgres service has not recovered. Redis on the same project went offline at the same time, and Postgres started dropping connections ("Connection terminated") with very slow disk I/O.

I tried restarting and redeploying the Postgres service, but the deployment never finishes. It stays stuck and the database is not accepting connections, so our production app is down.

Environment: production

Service: Postgres (with attached volume)

Region: US East

What I need:

  1. Please check whether the Postgres volume is on the failed host and move/restore it to healthy hardware.
  2. Please confirm the data on the volume is intact. If not, let me know the latest available backup.

I have not deleted or detached the volume. Please let me know if there is anything I should or should not do on my side.

Thank you!

Solved

1 Reply

Railway
BOT

19 days ago

The server your Postgres service ran on had hardware failures today at around 15:52 UTC and again at around 17:38 UTC, and our recovery work for both has finished. Your current Postgres deployment, started at 22:19 UTC, has been waiting since then for its container to start, so none of your code, variables, or settings is involved yet.

We restore affected services automatically, and occasionally one does not come back with it. To clear this, cancel that stuck deployment, then open the Postgres service, press CMD + K (CTRL + K on Windows and Linux), and run "Redeploy source image". Use the command palette rather than the Redeploy action on the existing deployment, since that re-runs the same deployment instead of starting a fresh one.

Redeploying does not delete, reset, or roll back the data on your volume, and keeping the volume attached as you have is correct. A hardware failure makes a volume unavailable rather than erasing it, and Postgres runs its own crash recovery on startup to return to a consistent state.

For mission-critical workloads, we advise implementing HA configs when possible, such as a primary and replica Postgres setup, so a single server failure does not take production down.


Status changed to Awaiting User Response Railway • 19 days ago


Status changed to Solved felipevt • 19 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...