postagres is broken after security update
pufaika
HOBBYOP

2 hours ago

URGENT: Production PostgreSQL service will not start — critical data on existing volume

Hello Railway Support,

My production PostgreSQL service is currently completely unavailable and stuck in a crash loop.

The issue began around the recent PostgreSQL security update. The database was working normally before this. I have not intentionally changed the PostgreSQL container configuration or deleted/recreated the database.

The deployment repeatedly shows:

ERROR (catatonit:2): failed to exec pid1: No such file or directory

The volume itself appears to mount successfully before the container crashes, for example:

Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/.../vol_0knh8m3j9glts8s6

I have already attempted the recovery steps previously recommended for this type of issue:

Restart / Redeploy

Redeploy Source Image

Migrated the PostgreSQL service and attached volume to US East

After migrating to US East, the exact same error continues:

ERROR (catatonit:2): failed to exec pid1: No such file or directory

This is a production database and it contains critical data that I cannot afford to lose.

PLEASE DO NOT:

wipe the volume

delete the database

recreate the database

reprovision it with an empty volume

reset PostgreSQL data

replace the existing data volume

My priority is preservation and recovery of the existing PostgreSQL data.

Please inspect the PostgreSQL service/container image, its entrypoint/PID1 configuration, and the attached volume from Railway's side and restore the service using the existing database files.

Please also verify that the complete original volume data was preserved during the region migration.

Project details:

Project ID:

c3802c2c-13e2-4f85-8cb8-b7cdf385063f

PostgreSQL Service ID:

09693d79-873d-4b79-9908-ed95d78ef8aa

Environment ID:

c0c16b3e-9362-42a9-9ee9-a7b73f249fa2

Current Volume ID after migration:

vol_0knh8m3j9glts8s6

Previous Volume ID before migration:

vol_namwoezarwc0s94v

PostgreSQL service URL:

https://railway.com/project/c3802c2c-13e2-4f85-8cb8-b7cdf385063f/service/09693d79-873d-4b79-9908-ed95d78ef8aa/database?environmentId=c0c16b3e-9362-42a9-9ee9-a7b73f249fa2

This appears to be a Railway container/runtime issue rather than an application-level error because PostgreSQL never reaches its normal startup process; catatonit fails while trying to execute PID1.

Please investigate this from the infrastructure side and let me know what is required to recover the existing database without data loss.

Thank you.

Solved

1 Replies

Railway
BOT

2 hours ago

Your volume data is intact (about 219 MB on the attached volume), so there is no data loss here. The crash is caused by the service's custom start command, which is currently set to an SQL statement rather than a valid executable. PostgreSQL's container entrypoint cannot run an SQL string as a process, which produces the catatonit: failed to exec pid1 error. Remove the start command from the service's settings (Settings > Deploy > Start Command) and redeploy. PostgreSQL will use its default entrypoint and boot normally against your existing data.


Status changed to Awaiting User Response Railway about 2 hours ago


Status changed to Solved pufaika about 2 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...