Production Postgres still failing after status page marked recovered — catatonit failed to exec pid1
hugmy-cs
HOBBYOP

a month ago

Hi Railway team,

Following the incident I23M92U0 (Google Cloud account block on May 19–20), the status page marks the disruption as resolved at 06:14 UTC on May 20, with a note that "some workloads may require redeployment."

I pressed Redeploy on my production Postgres service at approximately 06:18 UTC, but the container is still stuck in the exact same restart loop. Latest failure timestamp: 2026-05-20T06:18+ UTC (ongoing).

Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/8c6c858e-1eb6-44a6-babf-28ac3d99346a/vol_5sdpzv2zulhvbc8a

ERROR (catatonit:2): failed to exec pid1: No such file or directory

The replica ID has not changed across multiple redeploys, which suggests the container image is not being re-pulled. Without a fresh image, the loop cannot resolve itself.

Service Details

Project ID: 6b92ac7d-6881-4de1-af31-cd92b1b97bcb

Service ID (Postgres): db859e58-971f-4de6-96bc-0eef4190008c

Deployment ID: f8e83b14-c509-4751-a410-8a4b986a04c0

Replica ID: 8c6c858e-1eb6-44a6-babf-28ac3d99346a (unchanged across attempts)

Volume: vol_5sdpzv2zulhvbc8a

Plan: Hobby

Account: hugmy-cs

What I've already tried

Multiple Redeploy attempts via the dashboard — same error each time.

Request

Could you please:

Force a fresh container image pull for this service, or

Provide guidance on how to attach the existing volume (vol_5sdpzv2zulhvbc8a) to a newly created Postgres service so I can recover the data without losing what's on the volume.

The data on the volume is intact (mount succeeds every time); only the runtime image appears to be corrupted/missing. This is our production database, so we'd appreciate prompt assistance.

Thank you,

Taku

Solved

3 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


a month ago

Thanks for reaching out. We sincerely apologize for the service disruption.

We're seeing recovery in our API, builds, and deployments. If your service is experiencing an issue, please try redeploying it. We'll publish a public postmortem once we're fully recovered.

You can follow updates here: https://status.railway.com


Status changed to Awaiting User Response Railway about 1 month ago


hugmy-cs
HOBBYOP

a month ago

Hi,

Thanks for the quick response. Unfortunately, I've already tried Redeploy several times via the dashboard and the container continues to fail with the same error:

ERROR (catatonit:2): failed to exec pid1: No such file or directory

The replica ID has not changed across redeployments (8c6c858e-1eb6-44a6-babf-28ac3d99346a), which suggests the standard Redeploy button isn't pulling a fresh container image.

For reference, another user reported the exact same error earlier today and it was resolved when a Railway employee (mykal) manually redeployed the linked services on the backend:

https://station.railway.com/questions/postgres-container-failing-to-exec-pid1-4403449d

Could someone from the team do the same kind of backend redeploy for our Postgres service?

Service Details

Project ID: 6b92ac7d-6881-4de1-af31-cd92b1b97bcb

Service ID (Postgres): db859e58-971f-4de6-96bc-0eef4190008c

Replica ID: 8c6c858e-1eb6-44a6-babf-28ac3d99346a (unchanged across attempts)

Volume: vol_5sdpzv2zulhvbc8a (mount succeeds, data intact)

The data on the volume is intact — only the runtime image appears to be the issue. This is our production database, so any help would be appreciated.

Thanks,

Taku


Status changed to Awaiting Railway Response Railway about 1 month ago


hugmy-cs

Hi, Thanks for the quick response. Unfortunately, I've already tried Redeploy several times via the dashboard and the container continues to fail with the same error: ERROR (catatonit:2): failed to exec pid1: No such file or directory The replica ID has not changed across redeployments (8c6c858e-1eb6-44a6-babf-28ac3d99346a), which suggests the standard Redeploy button isn't pulling a fresh container image. For reference, another user reported the exact same error earlier today and it was resolved when a Railway employee (mykal) manually redeployed the linked services on the backend: → https://station.railway.com/questions/postgres-container-failing-to-exec-pid1-4403449d Could someone from the team do the same kind of backend redeploy for our Postgres service? Service Details Project ID: 6b92ac7d-6881-4de1-af31-cd92b1b97bcb Service ID (Postgres): db859e58-971f-4de6-96bc-0eef4190008c Replica ID: 8c6c858e-1eb6-44a6-babf-28ac3d99346a (unchanged across attempts) Volume: vol_5sdpzv2zulhvbc8a (mount succeeds, data intact) The data on the volume is intact — only the runtime image appears to be the issue. This is our production database, so any help would be appreciated. Thanks, Taku

a month ago

Hi Taku,

I've redeployed your service, and it seems to have come back online now. If your other services are having an issue, please try to give it a redeploy. We'll publish a public postmortem covering what happened when we're fully recovered.

For future reference, you can redeploy these services yourself in the same way I have been. Please let me know if you're still having issues.


Status changed to Awaiting User Response Railway about 1 month ago


Status changed to Solved mykal about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...