a month ago
Subject: Postgres Database Crashed - Certificate Permission Error - Data Recovery Needed
Project ID: 522542e9-724d-485a-b8e8-3aad131e0ad1
Environment: production
Service: Postgres
Issue Description:
Our Postgres database service has crashed and is unable to restart. The service shows a certificate permission error preventing startup.
Error Details:
FATAL: private key file "/var/lib/postgresql/data/certs/server.key" must be owned by the database user or root
LOG: database system is shut down
Affected Resources:
Service: Postgres
Deployment ID: 16cd2d7f-8c7a-4ce9-a60a-5621b95a6093
Volume ID: 252b2bb0-5be7-49a5-9caf-d568deabd931 (postgres-volume)
Volume Size: 5000 MB
Current Status:
Postgres service is in CRASHED state (crashed ~4 hours ago)
Replica status: 0 running, 1 crashed out of 1 total
Multiple restart attempts have failed with the same error
The volume appears to have corrupted certificate files
Data Recovery: We have daily backups available and would like to recover the database from the most recent backup rather than lose the data.
Request:
Can you check if there are volume snapshots available that we can restore from before the corruption occurred?
If snapshots are available, can you restore the volume to a previous state?
Alternatively, can you provide guidance on how to recover the data from the corrupted volume?
If recovery isn't possible, we can wipe the volume and restore from our daily backup manually.
Additional Context:
The backend service (jclb-go-backend) is also affected and waiting for the database to come online
We have daily backups that we can use for recovery if needed
Pinned Solution
a month ago
Try setting the start command to bash -c "chown postgres:postgres /var/lib/postgresql/data/certs/server.key && chmod 600 /var/lib/postgresql/data/certs/server.key" and redeploy the service. Once it’s successfully redeployed and online, remove the custom start command and redeploy again.
3 Replies
a month ago
Your Postgres service is crash-looping because the volume is mounted as root, but the Postgres process runs as a non-root user, so it cannot read the SSL certificate files on the volume. This is a known volume permissions issue, and you can fix it without restoring from backup by adding the environment variable RAILWAY_RUN_UID=0 to your Postgres service's variables, then redeploying. This tells the container to run as root, matching the volume's ownership, and Postgres should start normally with your existing data intact. Regarding backups, if you have volume backups configured, you can find and restore them from the Backups tab on the Postgres service.
Status changed to Awaiting User Response Railway • 26 days ago
Railway
Your Postgres service is crash-looping because the volume is mounted as root, but the Postgres process runs as a non-root user, so it cannot read the SSL certificate files on the volume. This is a known volume permissions issue, and you can fix it without restoring from backup by adding the environment variable `RAILWAY_RUN_UID=0` to your Postgres service's variables, then redeploying. This tells the container to run as root, matching the volume's ownership, and Postgres should start normally with your existing data intact. Regarding backups, if you have [volume backups](https://docs.railway.com/volumes/backups) configured, you can find and restore them from the Backups tab on the Postgres service.
a month ago
he RAILWAY_RUN_UID=0 fix from support isn't working. Postgres is still crashing with the certificate error. You need to contact support again and tell them:
The RAILWAY_RUN_UID=0 variable was added but Postgres is still crashing
Same error: private key file must be owned by the database user or root
The fix didn't resolve the issue
Status changed to Awaiting Railway Response Railway • 26 days ago
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 26 days ago
a month ago
Try setting the start command to bash -c "chown postgres:postgres /var/lib/postgresql/data/certs/server.key && chmod 600 /var/lib/postgresql/data/certs/server.key" and redeploy the service. Once it’s successfully redeployed and online, remove the custom start command and redeploy again.
Status changed to Solved sam-a • 26 days ago