Unable to rotate Postgres password for production database
jingle-bells-net
PROOP

3 months ago

Hello Railway Support,

I’m using Railway Postgres in production and I’d like to safely rotate the database credentials.

I already have PITR enabled, a recent Railway backup, and an external pg_dump backup stored safely.

When I tried using the “Regenerate Password” option under Postgres → Database → Config, I got this error:

“Failed to regenerate password — Password reset is not supported for Postgres HA clusters.”

I also noticed that manually editing POSTGRES_PASSWORD shows a warning saying it may only change the environment variable, not the actual database password, so I don’t want to do that and risk breaking the application.

Could you please let me know the safest supported way to rotate the PostgreSQL credentials for this service?

This is a production database, so my main concern is avoiding data loss and making sure the connected web service can reconnect properly after the change.

Thank you.

Solved

11 Replies

Railway
BOT

3 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 3 months ago


Click on the big Postgres HA group, go to the Database, tab, and you'll be able to regenerate the credentials from the Config tab.


jingle-bells-net
PROOP

3 months ago

I followed the suggested path exactly:

Postgres service → Database → Config → Regenerate Password

But the same error still appears:

“Failed to regenerate password — Password reset is not supported for Postgres HA clusters.”

The application is still healthy, but I need a supported way to rotate the Postgres credentials for this HA cluster without data loss and without breaking the connected web service.

Can you confirm the correct procedure for rotating credentials on a Railway Postgres HA cluster?


Status changed to Awaiting Railway Response Railway • 3 months ago


I'm checking with the team about this.


0x5b62656e5d

I'm checking with the team about this.

3 months ago

Have you checked it yet?


0x5b62656e5d

I'm checking with the team about this.

jingle-bells-net
PROOP

3 months ago

Hi, any update on this?

This is impacting a production application, so I need a clearer status to decide whether I should wait for Railway or apply a workaround on my side.

Could you please confirm if the team has already checked it, whether this is a known Railway issue, and if there is any recommended action or workaround?


jingle-bells-net

Hi, any update on this? This is impacting a production application, so I need a clearer status to decide whether I should wait for Railway or apply a workaround on my side. Could you please confirm if the team has already checked it, whether this is a known Railway issue, and if there is any recommended action or workaround?

3 months ago

Since password reset is not supported at railway dashboard, I think you should manually handle passwords.

Connect with a local database using psql or pgAdmin to update credentials.

Connect your database to pgAdmin.

Then open the query tool to execute the alter role with password command to update your password manually


3 months ago

پایگاه چجوری وصل شم



jingle-bells-net

Hi, any update on this? This is impacting a production application, so I need a clearer status to decide whether I should wait for Railway or apply a workaround on my side. Could you please confirm if the team has already checked it, whether this is a known Railway issue, and if there is any recommended action or workaround?

You'll need to do this manually.

  1. Click into the HA cluster
  2. Select the HA node (you might need to press "Show all services")
  3. Go to the Console tab
  4. Run psql, then ALTER ROLE postgres WITH PASSWORD '<NEW_PASS>'; where <NEW_PASS> is the new password
  5. Update the POSTGRES_PASSWORD variable on the original Postgres node you created the HA cluster from (IIRC it wouldn't have a number on it)
  6. Redeploy all associated services (including all Postgres nodes and the HA instance)

0x5b62656e5d

You'll need to do this manually. 1. Click into the HA cluster 2. Select the HA node (you might need to press "Show all services") 3. Go to the Console tab 4. Run `psql`, then `ALTER ROLE postgres WITH PASSWORD '<NEW_PASS>';` where `<NEW_PASS>` is the new password 5. Update the `POSTGRES_PASSWORD` variable on the original Postgres node you created the HA cluster from (IIRC it wouldn't have a number on it) 6. Redeploy all associated services (including all Postgres nodes and the HA instance)

jingle-bells-net
PROOP

3 months ago

Thanks for the instructions. Before I proceed on this production HA cluster, could you please confirm:

  1. After running ALTER ROLE postgres WITH PASSWORD '<NEW_PASS>' on the original HA Postgres node, will updating POSTGRES_PASSWORD automatically update the DATABASE_URL references used by the web service, or do I need to update any additional variable manually?

  2. What is the exact recommended redeploy order for:

    • the original Postgres node;
    • the Postgres replicas;
    • the HA instance / HAProxy;
    • the web service?
  3. Should I expect any downtime or replication interruption during this password rotation?

I want to avoid a password mismatch between the PostgreSQL role, POSTGRES_PASSWORD, the HA nodes, and the web service DATABASE_URL.


jingle-bells-net

Thanks for the instructions. Before I proceed on this production HA cluster, could you please confirm: 1. After running ALTER ROLE postgres WITH PASSWORD '<NEW_PASS>' on the original HA Postgres node, will updating POSTGRES_PASSWORD automatically update the DATABASE_URL references used by the web service, or do I need to update any additional variable manually? 2. What is the exact recommended redeploy order for: - the original Postgres node; - the Postgres replicas; - the HA instance / HAProxy; - the web service? 3. Should I expect any downtime or replication interruption during this password rotation? I want to avoid a password mismatch between the PostgreSQL role, POSTGRES_PASSWORD, the HA nodes, and the web service DATABASE_URL.

Step 5 handles your question 1.

For 2, redeploy all Postgres nodes (primary, then replicas), then HAProxy, then your web service. (At least that's what I did, and it worked)


Status changed to Solved 0x5b62656e5d • 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...