Old Postgres service was corrupted during outage, then railway recommended delete of container without backup
stillrollingcode
PROOP

12 days ago

Project ID: 66e2de6d-c5d9-4da9-8417-9253f39cf361

Environment: production

Issue: Old Postgres service was deleted; need to recover data from the old postgres-volume or check for backups

Timeline: Volume was mounted to new Postgres-Xj2V service on 2026-05-20, but password mismatch prevents access

Solved$20 Bounty

5 Replies

stillrollingcode
PROOP

12 days ago

If you could help me access the volume password I could reconnect my data. Rebuilding the database from scratch will take time and cost money, I am trying to avoid that.


Railway
BOT

12 days ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open Railway 12 days ago


Try this:

  1. Disable all public networking on the database if you have any, as the following steps will disable user authentication
  2. SSH into your database service (right click your service and select Copy SSH Command)
  3. Run this command: sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will bypass user authentication)
  4. Redeploy your database
  5. SSH again, and run the command psql
  6. Run ALTER USER postgres with password '<PASSWORD>'; where <PASSWORD> is the value of the variable PGPASSWORD in your Railway dashboard
  7. Type exit
  8. Run sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf (This will re-enable user authentication)
  9. Redeploy your database

0x5b62656e5d

Try this: 1. Disable all public networking on the database if you have any, as the following steps will disable user authentication 2. SSH into your database service (right click your service and select `Copy SSH Command`) 3. Run this command: `sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will bypass user authentication) 4. Redeploy your database 5. SSH again, and run the command `psql` 6. Run `ALTER USER postgres with password '<PASSWORD>';` where `<PASSWORD>` is the value of the variable `PGPASSWORD` in your Railway dashboard 7. Type `exit` 8. Run `sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will re-enable user authentication) 9. Redeploy your database

aayankali
FREE

12 days ago

hey totally different question how long does it take for bounty answer shift from pending approval to approved and when do we get bounty money

thanks


gaffarab99
FREE

12 days ago

The Problem

The old Postgres volume has data initialized with the old password, but the new PostgresXj2V service was created with new/different credentials — so Postgres refuses to start or authenticate properly against the existing data directory.

Solution (Step by Step)

Step 1 — Find the old password

Check anywhere the old DATABASE_URL was stored:

Your app's Railway environment variables (check version history)

Any .env files in your codebase/repo

Any other services in the project that had DATABASE_URL set — copy the password from it

Step 2 — Override the new service's password

In Railway, go to PostgresXj2V → Variables and manually set:

POSTGRES_PASSWORD=<your_old_password>

Make sure it exactly matches what the old service used. The data directory on the volume was initialized with that password, and Postgres will only unlock it with the same one.

Step 3 — Redeploy

Trigger a redeploy of PostgresXj2V. Since the volume is already mounted with the old data directory, Postgres should now start successfully and your data will be intact.

Step 4 — Immediately dump your data

Once you're in, don't wait:

bash

pg_dump U postgres d <your_db_name> > backup.sql

If you've lost the old password (fallback)

Use Railway's shell/exec into the container and do a trustauth bypass:

  1. Edit /var/lib/postgresql/data/pg_hba.conf — change scramsha256 to trust

  2. Reload: pg_ctl reload

  3. Connect without password and run:

    sql

    ALTER USER postgres WITH PASSWORD 'newpassword';

  4. Revert pg_hba.conf back to scramsha256

  5. Update the Railway env var POSTGRES_PASSWORD to match

The password mismatch is the only thing blocking you — the data is intact on the volume. Fix the credentials and you're in.


sam-a
EMPLOYEE

12 days ago

Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.

It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.

You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY

If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c

Feel free to respond if your question has not been addressed.


Status changed to Awaiting User Response Railway 12 days ago


Railway
BOT

5 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 5 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...