Postgres down 2h+ after hardware failure notice — two redeploys FAILED (project accurate-intuition)
mohakmoh87-hue
HOBBYOP

2 months ago

Our production database has been down for more than 2 hours. The 60-minute ETA in your outage email has long passed. This is a live business system (ISP subscriber management used by 3 offices) and it is completely unusable.

Project: accurate-intuition (9f5bbeb9-12d4-47b3-a0d5-520d8fd12f37)

Environment: production | Service: Postgres | Volume: postgres-volume | Region: US West (California)

Timeline (UTC):

22:40 Aug 20 - last healthy moment of the database

22:56 Aug 20 - app starts failing, every query returns ETIMEDOUT (Prisma: Can't reach database)

23:03 Aug 20 - redeploy attempt #1 -> FAILED

23:12 Aug 20 - your email: "hardware failure ... expected resolution within 60 minutes"

00:07 Aug 21 - redeploy attempt #2 -> FAILED (only log line: "Stopping Container")

01:00 Aug 21 - still down

The app service itself is perfectly healthy; only the database is unreachable, so the volume still appears to be attached to the failed host. I also see many other users reporting the same thing here right now.

Requests:

  1. Please escalate and move postgres-volume to a healthy host as soon as possible.

  2. Please confirm the volume data is intact and not at risk.

  3. Please give a realistic ETA, so I can decide whether to restore from my own external backup onto a fresh database.

Note: we are on the Hobby plan, so Railway-side backups/PITR are not available to us. An accurate ETA really matters here.

Thank you.

Solved

1 Replies

Status changed to Awaiting Railway Response Railway • about 2 months ago


dizzydes90
EMPLOYEE

2 months ago

Unfortunately we're having an incident with the host you're project is currently on, recovery is starting and expected in the next hour.

Please accept my apologies we have several platform engineers investigating this right now. Will update here as I know more.


Status changed to Awaiting User Response Railway • about 2 months ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...