a month ago
Hi Railway Team,
My production application has been DOWN for 5+ hours due to a Postgres authentication issue I cannot resolve from the dashboard. Real customers affected.
## Project Details
- Project ID: 243bc37a-7017-4969-89c0-fe676bc7f663
- Environment: production
- Service with issue: facturate-sv (Django/Python app)
- Database: Postgres (same project)
## The Problem
All connection attempts from facturate-sv to Postgres fail repeatedly with:
FATAL: password authentication failed for user "postgres"
Postgres service shows Online, but authentication is completely broken.
## What Triggered It
Earlier today I rotated POSTGRES_PASSWORD manually from the Postgres Variables tab in the Railway dashboard. Immediately after, linked services could no longer authenticate.
## What I Tried (nothing worked)
1. Re-linked DATABASE_URL in facturate-sv as a reference to Postgres
2. Restarted facturate-sv multiple times (Restart and Redeploy)
3. Added a FORCE_RESTART dummy variable to Postgres to trigger a restart
4. Reverted all recent code changes via git revert back to yesterday's working version
5. Multiple redeploys with fresh builds
The error persists identically in all attempts. The Postgres IP address has changed (fd12:ca1b:4dfe:1:2000:3c:fb4:9553 → fd12:ca1b:4dfe:1:2000:3c:bb4a:9103) confirming the service has restarted, but auth still fails.
This confirms the issue is with Postgres credentials being out of sync INTERNALLY at the server level, not with application code or variable references.
## Business Impact
Real paying customers (small businesses in El Salvador using electronic invoicing, which is legally mandatory) have been unable to operate for 5+ hours. This is severely urgent.
## What I Need
Could you please:
1. Resynchronize the Postgres credentials so POSTGRES_PASSWORD matches what the database server is actually using, OR
2. Provide the correct recovery procedure for this state
I am available immediately to answer any questions or provide additional logs.
Thank you very much for your help.
1 Replies
a month ago
Manually editing POSTGRES_PASSWORD in the Variables tab only changes the environment variable, not the actual password stored inside the Postgres database (the init script only runs on first boot when the volume is empty), which is why authentication is failing. To fix this, open your Postgres service, go to the Credentials tab in the database view, and use the regenerate password button there - this safely updates both the database password and the environment variable in sync. After regenerating, redeploy any dependent services like facturate-sv so they pick up the new DATABASE_URL. More details in our docs.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 29 days ago