Project: restaurant-analysis (106ff76f-5655-4723-a328-d38e691fd237) Environment: production (4db1ff34-397b-43e5-b4cf-faa0e9d0931e) Postgres Service: postgres-production-3438 (611a8eb9-d672-40db-b332-aa96e0636254) App Service: restaurant-analysis (87c63d05-a8c3-400c-b1f0-217af1d1217d) URGENT: production app is down. The postgres user's password got out of sync with all environment variables after multiple ALTER USER attempts today. - Postgres Online, data intact - restaurant-analysis CRASHED: "FATAL: password authentication failed for user postgres" - All external connections rejected; no in-container terminal access - All DB variables are direct values Request (either is fine): (A) If possible, roll back the Postgres password / variables to the state before today's changes (before ~03:00 UTC / 12:00 JST July 9). (B) Otherwise, reset the postgres user's password at the infrastructure level (data intact) and set POSTGRES_PASSWORD / PGPASSWORD / DATABASE_URL / DATABASE_PUBLIC_URL to a single matching value.

Try this: 1. Disable all public networking on the database if you have any, as the following steps will disable user authentication 2. SSH into your database service (right click your service and select `Copy SSH Command`) 3. Run this command: `sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will bypass user authentication) 4. Redeploy your database 5. SSH again, and run the command `psql` 6. Run `ALTER USER postgres with password ' ';` where ` ` is the value of the variable `PGPASSWORD` in your Railway dashboard 7. Type `exit` 8. Run `sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will re-enable user authentication) 9. Redeploy your database

Postgres password out of sync - production app down

suirunakamura

PROOP

3 months ago

Project: restaurant-analysis (106ff76f-5655-4723-a328-d38e691fd237)

Environment: production (4db1ff34-397b-43e5-b4cf-faa0e9d0931e)

Postgres Service: postgres-production-3438 (611a8eb9-d672-40db-b332-aa96e0636254)

App Service: restaurant-analysis (87c63d05-a8c3-400c-b1f0-217af1d1217d)

URGENT: production app is down. The postgres user's password got out of sync

with all environment variables after multiple ALTER USER attempts today.

Postgres Online, data intact
restaurant-analysis CRASHED: "FATAL: password authentication failed for user postgres"
All external connections rejected; no in-container terminal access
All DB variables are direct values

Request (either is fine):

(A) If possible, roll back the Postgres password / variables to the state

  before today's changes (before ~03:00 UTC / 12:00 JST July 9).

(B) Otherwise, reset the postgres user's password at the infrastructure level

  (data intact) and set POSTGRES_PASSWORD / PGPASSWORD / DATABASE_URL / 

  DATABASE_PUBLIC_URL to a single matching value.

Solved$20 Bounty

Pinned Solution

0x5b62656e5d

MODERATOR

3 months ago

Try this:

Disable all public networking on the database if you have any, as the following steps will disable user authentication
SSH into your database service (right click your service and select Copy SSH Command)
Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)
Redeploy your database
SSH again, and run the command psql
Run ALTER USER postgres with password '<PASSWORD>'; where <PASSWORD> is the value of the variable PGPASSWORD in your Railway dashboard
Type exit
Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)
Redeploy your database

3 Replies

Railway

BOT

3 months ago

The Credentials tab on your Postgres service can regenerate the password and update all related environment variables in sync, which is exactly what you need here. Open the Postgres service, go to the Credentials tab, and use the regenerate option. After that, redeploy your restaurant-analysis service so it picks up the new values.

Status changed to Awaiting User Response Railway • 3 months ago

suirunakamura

PROOP

3 months ago

The bot's answer doesn't work — there is NO "Credentials" tab on the Postgres

service in the current UI. I checked all tabs:

Deployments / Database / Backups / Variables / Metrics / Console / Settings.

Under Settings: Source / Networking / Edge / Scale / Deploy / Feature-flags / Danger.

There is no self-service password regenerate option anywhere.

I have already confirmed (via Railway's own AI assistant) that this requires

infrastructure-level access. Please have a human from the Railway team:

(A) roll back the Postgres password/variables to before ~03:00 UTC July 9, OR

(B) reset the postgres user's password at the infra level (data intact) and

sync POSTGRES_PASSWORD / PGPASSWORD / DATABASE_URL / DATABASE_PUBLIC_URL 

to a single matching value.

Production app (restaurant-analysis) has been down for hours. This is urgent.

Status changed to Awaiting Railway Response Railway • 3 months ago

Railway

BOT

3 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 3 months ago

0x5b62656e5d

MODERATOR

3 months ago

Try this:

Disable all public networking on the database if you have any, as the following steps will disable user authentication
SSH into your database service (right click your service and select Copy SSH Command)
Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)
Redeploy your database
SSH again, and run the command psql
Run ALTER USER postgres with password '<PASSWORD>'; where <PASSWORD> is the value of the variable PGPASSWORD in your Railway dashboard
Type exit
Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)
Redeploy your database

Status changed to Solved 0x5b62656e5d • 7 days ago

Welcome!