a month ago
Hi Railway team,
I'm experiencing what appears to be a known bug with Postgres password synchronization on my project. Here are the details:
Project: SistemaAmbiental
Services:web (FastAPI on app.hlarreta.com) + Postgres (with postgres-volume)
Issue:
Yesterday I attempted to rotate my Postgres password by updating the POSTGRES_PASSWORD variable on the Postgres service. After the restart, my web service could no longer authenticate with the error:
```
FATAL: password authentication failed for user "postgres"
connection to server at "postgres.railway.internal" port 5432 failed
```
The Postgres service shows as Online, but the actual password stored in the postgres role inside the database appears to be OUT OF SYNC with the POSTGRES_PASSWORD environment variable. I verified this because:
1. Connecting with the value in POSTGRES_PASSWORD fails with auth error
2. Connecting with the PREVIOUS password (before rotation) still works
3. DATABASE_URL and DATABASE_PUBLIC_URL variables did update to reflect the new password, but the database role itself did not
Current status:
My web service has been crashed for 12+ hours. I rolled back POSTGRES_PASSWORD to the previous value as a workaround, but the password is now compromised (it was exposed in logs during debugging).
What I need:
1. Confirmation of this bug and a safe procedure to rotate the postgres user password without breaking the service
2. Ideally, Railway performs the rotation server-side so both the role password AND the env variable stay in sync
3. Guidance on whether ALTER USER postgres WITH PASSWORD '...' via psql is the correct manual workaround
Happy to share project ID or any other info needed via DM. This is blocking a production deployment for an accredited environmental laboratory (real business impact).
Thanks,
Humberto Larreta
Hlarreta Laboratorio Ambiental
Pinned Solution
a month ago
Try this:
1. Disable all public networking on the database if you have any, as the following steps will disable user authentication
2. SSH into your database service (right click your service and select Copy SSH Command)
3. Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)
4. Redeploy your database
5. SSH again, and run the command psql
6. Run ALTER USER postgres with password '<PASSWORD>'; where <PASSWORD> is the value of the variable PGPASSWORD in your Railway dashboard
7. Type exit
8. Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)
9. Redeploy your database
2 Replies
Status changed to Open Railway • 28 days ago
a month ago
Try this:
1. Disable all public networking on the database if you have any, as the following steps will disable user authentication
2. SSH into your database service (right click your service and select Copy SSH Command)
3. Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)
4. Redeploy your database
5. SSH again, and run the command psql
6. Run ALTER USER postgres with password '<PASSWORD>'; where <PASSWORD> is the value of the variable PGPASSWORD in your Railway dashboard
7. Type exit
8. Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)
9. Redeploy your database
a month ago
Keep in mind that when you wish to change your Postgres password, you must do so via the database panel > configs, where you'll find a button that says "Regenerate."
Status changed to Solved 0x5b62656e5d • 4 days ago