Database timeout
zmatheuspro
PROOP

2 days ago

Today, out of nowhere, all my apps timeout out with the database. The database is simply saying the password is invalid even though I haven't changed anything, and I can't do absolutely anything. I can't rotate credentials, I can't restart the container, I'm losing money.

29 Replies

Is this a Postgres database?


zmatheuspro
PROOP

2 days ago

yes


zmatheuspro
PROOP

2 days ago

I tried restarting the container but it froze at: Mounting volume on /var/lib/containers/railwayapp/bind-mounts/f0259a67-a79f-4ccd-981c-7cf9a648a21c/vol_a8qsn2en58a2l92t


Try redeploying the database instead of restarting it.


The database is simply saying the password is invalid
Does it say something like authentication failed for user postgres?


zmatheuspro
PROOP

2 days ago

It results in the same thing.


zmatheuspro
PROOP

2 days ago

2026-05-12 00:18:17.500 UTC [88280] FATAL: password authentication failed for user "postgres"
2026-05-12 00:18:17.500 UTC [88280] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256" 2026-05-12 00:18:20.779 UTC [88268] ERROR: canceling statement due to statement timeout 2026-05-12 00:18:20.779 UTC [88268] STATEMENT: /* railway:dataui / SET statementtimeout = '30s'; / railway:dataui */ SELECT tablename FROM informationschema.tables WHERE table_schema = 'public'
2026-05-12 00:18:20.780 UTC [88268] LOG: could not send data to client: Broken pipe
2026-05-12 00:18:20.780 UTC [88268] FATAL: connection to client lost


zmatheuspro
PROOP

2 days ago

That was the last log entry; after that, nothing else worked.


Try this:

  1. Disable all public networking on the database if you have any, as the following steps will disable user authentication

  2. SSH into your database service (right click your service and select Copy SSH Command)

  3. Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)

  4. Redeploy your database

  5. SSH again, and run the command psql

  6. Run ALTER USER postgres with password ''; where ` is the value of the variablePGPASSWORD` in your Railway dashboard

  7. Type exit

  8. Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)

  9. Redeploy your database


prodrigocostavieira
PRO

2 days ago

pls, i need help. I have a similar problem. A few hours ago, my database stopped responding. I made some backups but I can't restore them; I've tried but it didn't work. What do I do? I haven't changed anything at all; it was something sudden. I've already tried restoring the volume but without success; the data isn't recovered.

Attachments


Please create your own thread.


zmatheuspro
PROOP

2 days ago

📭 No active deployment for '1560038d-ea05-45df-8248-05994885fbe5'
🔧 Deploy your service first, then try again

Connection to ssh.railway.com closed by remote host.
Connection to ssh.railway.com closed.


zmatheuspro
PROOP

2 days ago

I can't connect because the container simply won't start.


Try adding a custom start command to be sleep infinity, then try SSH-ing again. Make sure to revert that after step 8.


zmatheuspro
PROOP

2 days ago

It's now stuck in QUEUED mode even though there are no active deployments.


zmatheuspro
PROOP

2 days ago

I left the line, I'll see.


zmatheuspro
PROOP

2 days ago

root@e446dfef3e1e:/# psql
psql: error: connection to server at "database.railway.internal" (ipv6), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?


zmatheuspro
PROOP

2 days ago

I managed to initialize the database and perform the alteration; I'll test it again.


zmatheuspro
PROOP

a day ago

Following all the steps, it worked now.


Good to know.


Anything else before I close the thread?


zmatheuspro
PROOP

a day ago

Just one additional note: when you run psql and get an error, you need to execute:
su - postgres -c "/usr/lib/postgresql/16/bin/pg_ctl start -D /var/lib/postgresql/data/pgdata"

This will make it work. If someone needs similar help, it's good to mention this.


I'll keep that in mind next time. Thanks!


zmatheuspro
PROOP

21 hours ago

It happened again out of nowhere.


zmatheuspro
PROOP

21 hours ago

2026-05-12 07:33:27.494 UTC [11845] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256" 2026-05-12 07:33:32.495 UTC [11847] FATAL: password authentication failed for user "postgres" 2026-05-12 07:33:32.495 UTC [11847] DETAIL: Connection matched file "/var/lib/postgresql/data/pgdata/pghba.conf" line 128: "host all all all scram-sha-256"


zmatheuspro
PROOP

20 hours ago

My Postgres volume was at 50 GB. I noticed this happened after the daily backup, and my database was already occupying 30 GB. I don't know if this interferes with anything, but I reduced it to 200 GB and repeated the process above, and it worked. Let's see if this happens again.


zmatheuspro
PROOP

20 hours ago

It's happening again, the app is still working but I saw that it went from 20 connections to 2, I really can't diagnose the problem.

Attachments


Try this thing again.


Looks like the password somehow broke itself…


Welcome!

Sign in to your Railway account to join the conversation.

Loading...