2 days ago
Today, out of nowhere, all my apps timeout out with the database. The database is simply saying the password is invalid even though I haven't changed anything, and I can't do absolutely anything. I can't rotate credentials, I can't restart the container, I'm losing money.
29 Replies
2 days ago
Is this a Postgres database?
I tried restarting the container but it froze at: Mounting volume on /var/lib/containers/railwayapp/bind-mounts/f0259a67-a79f-4ccd-981c-7cf9a648a21c/vol_a8qsn2en58a2l92t
2 days ago
Try redeploying the database instead of restarting it.
2 days ago
The database is simply saying the password is invalid
Does it say something like authentication failed for user postgres?
2026-05-12 00:18:17.500 UTC [88280] FATAL: password authentication failed for user "postgres"
2026-05-12 00:18:17.500 UTC [88280] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256" 2026-05-12 00:18:20.779 UTC [88268] ERROR: canceling statement due to statement timeout 2026-05-12 00:18:20.779 UTC [88268] STATEMENT: /* railway:dataui / SET statementtimeout = '30s'; / railway:dataui */ SELECT tablename FROM informationschema.tables WHERE table_schema = 'public'
2026-05-12 00:18:20.780 UTC [88268] LOG: could not send data to client: Broken pipe
2026-05-12 00:18:20.780 UTC [88268] FATAL: connection to client lost
2 days ago
Try this:
Disable all public networking on the database if you have any, as the following steps will disable user authentication
SSH into your database service (right click your service and select
Copy SSH Command)Run this command:
sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will bypass user authentication)Redeploy your database
SSH again, and run the command
psqlRun
ALTER USER postgres with password '';where` is the value of the variablePGPASSWORD` in your Railway dashboardType
exitRun
sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will re-enable user authentication)Redeploy your database
2 days ago
pls, i need help. I have a similar problem. A few hours ago, my database stopped responding. I made some backups but I can't restore them; I've tried but it didn't work. What do I do? I haven't changed anything at all; it was something sudden. I've already tried restoring the volume but without success; the data isn't recovered.
Attachments
2 days ago
Please create your own thread.
📭 No active deployment for '1560038d-ea05-45df-8248-05994885fbe5'
🔧 Deploy your service first, then try again
Connection to ssh.railway.com closed by remote host.
Connection to ssh.railway.com closed.
2 days ago
Try adding a custom start command to be sleep infinity, then try SSH-ing again. Make sure to revert that after step 8.
It's now stuck in QUEUED mode even though there are no active deployments.
root@e446dfef3e1e:/# psql
psql: error: connection to server at "database.railway.internal" (ipv6), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
I managed to initialize the database and perform the alteration; I'll test it again.
a day ago
Good to know.
a day ago
Anything else before I close the thread?
Just one additional note: when you run psql and get an error, you need to execute: su - postgres -c "/usr/lib/postgresql/16/bin/pg_ctl start -D /var/lib/postgresql/data/pgdata"
This will make it work. If someone needs similar help, it's good to mention this.
a day ago
I'll keep that in mind next time. Thanks!
2026-05-12 07:33:27.494 UTC [11845] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256" 2026-05-12 07:33:32.495 UTC [11847] FATAL: password authentication failed for user "postgres" 2026-05-12 07:33:32.495 UTC [11847] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256"
My Postgres volume was at 50 GB. I noticed this happened after the daily backup, and my database was already occupying 30 GB. I don't know if this interferes with anything, but I reduced it to 200 GB and repeated the process above, and it worked. Let's see if this happens again.
It's happening again, the app is still working but I saw that it went from 20 connections to 2, I really can't diagnose the problem.
Attachments
13 hours ago
Try this thing again.
13 hours ago
Looks like the password somehow broke itself…