a month ago
Hello,
I have a critical issue with my PostgreSQL database service that has been crashing since May 20, 2026, and I cannot establish any external connections.
Service Details:
Project ID: d8f00be0-636e-420c-bac5-9a745431cda6
Environment: homolog
Service Name: portal-alumni-v2-db
Deployment ID: 02fa72ba-8aca-4c23-9029-4d7038d9331a
Error Message: The service is repeatedly failing with the error:
ERROR (catatonit:2): failed to exec pid1: No such file or directory
The container keeps mounting the volume but fails to initialize the PostgreSQL process.
Connection Attempts: When I try to connect externally using the DATABASE_PUBLIC_URL provided by Railway, I receive:
FATAL: password authentication failed for user "postgres"
However, this appears to be because the database service cannot start, not an authentication issue.
Critical Constraint: This database contains production data that cannot be lost. Deletion is not an option.
Request: Please investigate:
Volume corruption or integrity issues
Availability of volume snapshots for recovery
Any way to restore the database from a previous state
I cannot create a backup because the service is unable to start and accept connections.
Thank you for your urgent assistance.
Pinned Solution
a month ago
the password in DATABASE_PUBLIC_URL is what railway has stored, but the postgres container's actual auth state got reset to the value of its POSTGRES_PASSWORD env var when the source image was re-pulled — they no longer match. classic auth desync after a fresh image pull.
fix is to force the database's stored password to match what railway has:
- disable any public domain on the database service first (this process temporarily disables auth)
- SSH into the database — right-click the service → copy SSH command, paste in your terminal (needs railway cli installed locally)
- run:
sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf - redeploy the database from the dashboard
- SSH back in, run
psql ALTER USER postgres WITH PASSWORD '<value of the PGPASSWORD env var on the postgres service>';- revert:
sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf - redeploy once more, re-enable the public domain if you had one
after step 8, DATABASE_PUBLIC_URL will authenticate. your data is intact throughout — this only changes the user's stored password, nothing in the actual tables.
5 Replies
a month ago
Your volume is intact and in a healthy state with ~1.4 GB of data. The "failed to exec pid1: No such file or directory" error is caused by a stale container image, not volume corruption. To fix this, open your service, press Cmd/Ctrl+K to open the command palette, and select "Redeploy source image" - this re-pulls a fresh image and resolves the error. A normal redeploy from the 3-dot menu will not work because it reuses the cached image.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
Your volume is intact and in a healthy state with ~1.4 GB of data. The "failed to exec pid1: No such file or directory" error is caused by a stale container image, not volume corruption. To fix this, open your service, press Cmd/Ctrl+K to open the command palette, and select "Redeploy source image" - this re-pulls a fresh image and resolves the error. A normal redeploy from the 3-dot menu will not work because it reuses the cached image.
a month ago
Hello,
Thank you for recovering the portal-alumni-v2 database service. The service is now online and responding, which is great progress.
However, I'm facing a critical credential issue:
The Problem: The DATABASE_PUBLIC_URL provided in the homolog environment variables is not authenticating:
postgresql://postgres:CbrNvWVZiFSxjXwxZjWNFtiPQTsizHnd@hopper.proxy.rlwy.net:33327/railway
Error received:
FATAL: password authentication failed for user "postgres"
The host and port are reachable (TCP connection established), but the password is being rejected.
Additional Context: I have two environments:
Production environment: Database connects successfully with its credentials
Homolog environment: Database is online but credentials don't work
The critical issue is that all my production data is in the homolog environment, and the production database is empty. I need to access the homolog database to restore the data.
Request: Please:
Verify the current password for the homolog portal-alumni-v2 database
Regenerate the credentials if necessary
Provide the correct DATABASE_PUBLIC_URL or PGPASSWORD that will authenticate against the recovered database
This is blocking my ability to access and backup critical production data.
Thank you for your urgent assistance.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Hello,
Thank you for recovering the portal-alumni-v2 database service. The service is now online and responding, which is great progress.
However, I'm facing a critical credential issue:
The Problem: The DATABASE_PUBLIC_URL provided in the homolog environment variables is not authenticating:
postgresql://postgres:CbrNvWVZiFSxjXwxZjWNFtiPQTsizHnd@hopper.proxy.rlwy.net:33327/railway
Error received:
FATAL: password authentication failed for user "postgres"
The host and port are reachable (TCP connection established), but the password is being rejected.
Additional Context: I have two environments:
Production environment: Database connects successfully with its credentials
Homolog environment: Database is online but credentials don't work
The critical issue is that all my production data is in the homolog environment, and the production database is empty. I need to access the homolog database to restore the data.
Request: Please:
Verify the current password for the homolog portal-alumni-v2 database
Regenerate the credentials if necessary
Provide the correct DATABASE_PUBLIC_URL or PGPASSWORD that will authenticate against the recovered database
This is blocking my ability to access and backup critical production data.
Thank you for your urgent assistance.
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
the password in DATABASE_PUBLIC_URL is what railway has stored, but the postgres container's actual auth state got reset to the value of its POSTGRES_PASSWORD env var when the source image was re-pulled — they no longer match. classic auth desync after a fresh image pull.
fix is to force the database's stored password to match what railway has:
- disable any public domain on the database service first (this process temporarily disables auth)
- SSH into the database — right-click the service → copy SSH command, paste in your terminal (needs railway cli installed locally)
- run:
sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf - redeploy the database from the dashboard
- SSH back in, run
psql ALTER USER postgres WITH PASSWORD '<value of the PGPASSWORD env var on the postgres service>';- revert:
sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf - redeploy once more, re-enable the public domain if you had one
after step 8, DATABASE_PUBLIC_URL will authenticate. your data is intact throughout — this only changes the user's stored password, nothing in the actual tables.
richwardle
the password in DATABASE_PUBLIC_URL is what railway has stored, but the postgres container's actual auth state got reset to the value of its `POSTGRES_PASSWORD` env var when the source image was re-pulled — they no longer match. classic auth desync after a fresh image pull. fix is to force the database's stored password to match what railway has: 1. disable any public domain on the database service first (this process temporarily disables auth) 2. SSH into the database — right-click the service → copy SSH command, paste in your terminal (needs railway cli installed locally) 3. run: `sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf` 4. redeploy the database from the dashboard 5. SSH back in, run `psql` 6. `ALTER USER postgres WITH PASSWORD '<value of the PGPASSWORD env var on the postgres service>';` 7. revert: `sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf` 8. redeploy once more, re-enable the public domain if you had one after step 8, DATABASE_PUBLIC_URL will authenticate. your data is intact throughout — this only changes the user's stored password, nothing in the actual tables.
a month ago
Hey, thanks a lot! That ticket was a lifesaver. The solution worked perfectly — database synced and authenticating smoothly now. 🙌
Status changed to Solved brody • about 1 month ago