2 months ago
Today, out of nowhere, all my apps timeout out with the database. The database is simply saying the password is invalid even though I haven't changed anything, and I can't do absolutely anything. I can't rotate credentials, I can't restart the container, I'm losing money.
29 Replies
2 months ago
Is this a Postgres database?
I tried restarting the container but it froze at: Mounting volume on /var/lib/containers/railwayapp/bind-mounts/f0259a67-a79f-4ccd-981c-7cf9a648a21c/vol_a8qsn2en58a2l92t
2 months ago
Try redeploying the database instead of restarting it.
2 months ago
The database is simply saying the password is invalid
Does it say something like authentication failed for user postgres?
2026-05-12 00:18:17.500 UTC [88280] FATAL: password authentication failed for user "postgres"
2026-05-12 00:18:17.500 UTC [88280] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all scram-sha-256"
2026-05-12 00:18:20.779 UTC [88268] ERROR: canceling statement due to statement timeout
2026-05-12 00:18:20.779 UTC [88268] STATEMENT: /* railway:dataui / SET statement_timeout = '30s'; / railway:dataui */ SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'
2026-05-12 00:18:20.780 UTC [88268] LOG: could not send data to client: Broken pipe
2026-05-12 00:18:20.780 UTC [88268] FATAL: connection to client lost
2 months ago
Try this:
- Disable all public networking on the database if you have any, as the following steps will disable user authentication
- SSH into your database service (right click your service and select
Copy SSH Command) - Run this command:
sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will bypass user authentication) - Redeploy your database
- SSH again, and run the command
psql - Run
ALTER USER postgres with password '';where `` is the value of the variablePGPASSWORDin your Railway dashboard - Type
exit - Run
sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will re-enable user authentication) - Redeploy your database
2 months ago
pls, i need help. I have a similar problem. A few hours ago, my database stopped responding. I made some backups but I can't restore them; I've tried but it didn't work. What do I do? I haven't changed anything at all; it was something sudden. I've already tried restoring the volume but without success; the data isn't recovered.
Attachments
2 months ago
Please create your own thread.
📭 No active deployment for '1560038d-ea05-45df-8248-05994885fbe5'
🔧 Deploy your service first, then try again
Connection to ssh.railway.com closed by remote host.
Connection to ssh.railway.com closed.
2 months ago
Try adding a custom start command to be sleep infinity, then try SSH-ing again. Make sure to revert that after step 8.
It's now stuck in QUEUED mode even though there are no active deployments.
root@e446dfef3e1e:/# psql
psql: error: connection to server at "database.railway.internal" (ipv6), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
I managed to initialize the database and perform the alteration; I'll test it again.
2 months ago
Good to know.
2 months ago
Anything else before I close the thread?
Just one additional note: when you run psql and get an error, you need to execute:
su - postgres -c "/usr/lib/postgresql/16/bin/pg_ctl start -D /var/lib/postgresql/data/pgdata"
This will make it work. If someone needs similar help, it's good to mention this.
2 months ago
I'll keep that in mind next time. Thanks!
2026-05-12 07:33:27.494 UTC [11845] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all scram-sha-256"
2026-05-12 07:33:32.495 UTC [11847] FATAL: password authentication failed for user "postgres"
2026-05-12 07:33:32.495 UTC [11847] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pg_hba.conf" line 128: "host all all all scram-sha-256"
My Postgres volume was at 50 GB. I noticed this happened after the daily backup, and my database was already occupying 30 GB. I don't know if this interferes with anything, but I reduced it to 200 GB and repeated the process above, and it worked. Let's see if this happens again.
It's happening again, the app is still working but I saw that it went from 20 connections to 2, I really can't diagnose the problem.
Attachments
2 months ago
Try this thing again.
2 months ago
Looks like the password somehow broke itself…