Skip to content
Regenerate disabled for sealed PostgreSQL and Redis passwords in staging and production
brunolliveira
OP

a month ago

I am completing credential rotation following the October 21 security incident, but Regenerate is disabled under Database → Config → Connection for PostgreSQL and Redis in both staging and production. No tooltip explains why.

Account and credentials:

  • Hobby plan, personal workspace.
  • Project role: Owner. Workspace role: Admin.
  • All four password variables are sealed: POSTGRES_PASSWORD and REDIS_PASSWORD in both environments.
  • I cannot view their current values.
  • Regenerate was previously used during an earlier security incident in August 2026.

The Railway agent suggested checking deployment activity, pending regeneration changes, permissions, and database-template eligibility. We checked the following on October 25, 2026:

Deployment activity: All four database services are online, with one running replica each and active deployments marked SUCCESS. No database deployment is currently building or deploying.

Pending changes: Both describe_environment and get_staged_changes return staged: null, with no resource or variable changes. However, environment_status.pendingWork reports an EnvironmentPatch marked staged with changes: [] in each environment. These records date from October 23. Could these empty patch records be blocking regeneration?

Service configuration: All four services have persistent volumes. Both PostgreSQL services use ghcr.io/railwayapp-templates/postgres-ssl:17; both Redis services use redis:8.2.9. Please confirm whether these services satisfy the eligibility checks for Regenerate.

No database passwords have been changed manually. Previous password rotations were performed exclusively through the Regenerate button, following Railway’s documented procedure.

Could you please confirm:

  1. Which exact condition is disabling Regenerate?
  2. Can sealed passwords or the empty pending-patch records cause this?
  3. Is there a cooldown or frequency limit after a previous regeneration?
  4. Are any additional permission, template-configuration, or platform restrictions applicable?
  5. What is the supported rotation procedure that preserves existing data and volumes and keeps database credentials, environment variables, and references synchronized?

I emailed support on October 23 and am still awaiting a response. I can provide project, service, and patch IDs and the previous email reference privately.

Awaiting Railway Response

5 Replies

Railway
BOT

a month ago

We can't confirm from here which condition is disabling Regenerate on these four services. We also can't confirm that sealed variables, the empty October 23 patch records, a cooldown, or a template or permission check is the cause, so we won't name one. Owner and Admin roles, healthy deployments, and volumes attached don't rule a cause in or out by themselves.

One open incident is relevant: its latest update says some non-essential product features remain disabled while service is being restored. That update doesn't list which features are affected, so we can't say whether Regenerate is one of them.

The supported way to rotate is still Regenerate in the database's Config tab under Connection. It changes the password inside the database and the password variable together, and your data and volume stay in place. After it runs, redeploy every service that references the password variable or the database URL, since existing connections fail until they use the new password (docs).

Until Regenerate is available, don't edit POSTGRES_PASSWORD or REDIS_PASSWORD directly, and don't run ALTER USER or a Redis equivalent. Each of those changes only one side and leaves the database and the variable out of sync, which causes authentication failures. Because these variables are sealed, their current values can't be read back to repair a mismatch afterwards.


Status changed to Awaiting User Response Railway • 26 days ago


Railway

We can't confirm from here which condition is disabling Regenerate on these four services. We also can't confirm that sealed variables, the empty October 23 patch records, a cooldown, or a template or permission check is the cause, so we won't name one. Owner and Admin roles, healthy deployments, and volumes attached don't rule a cause in or out by themselves. One open incident is relevant: its latest update says [some non-essential product features remain disabled](https://status.railway.com/incident/ZEO9X68O) while service is being restored. That update doesn't list which features are affected, so we can't say whether Regenerate is one of them. The supported way to rotate is still Regenerate in the database's Config tab under Connection. It changes the password inside the database and the password variable together, and your data and volume stay in place. After it runs, redeploy every service that references the password variable or the database URL, since existing connections fail until they use the new password ([docs](https://docs.railway.com/databases/database-view)). Until Regenerate is available, don't edit POSTGRES_PASSWORD or REDIS_PASSWORD directly, and don't run ALTER USER or a Redis equivalent. Each of those changes only one side and leaves the database and the variable out of sync, which causes authentication failures. Because these variables are sealed, their current values can't be read back to repair a mismatch afterwards.

brunolliveira
OP

a month ago

Thank you for clarifying what remains unconfirmed.

Could a Railway engineer review the four affected database services and identify the exact condition disabling Regenerate?

In particular, please check:

  1. Whether Regenerate is intentionally disabled as part of incident ZEO9X68O.
  2. Whether the empty October 23 EnvironmentPatch records reported as staged could be blocking the action, despite the staged-changes queries returning no changes.
  3. If neither applies, which service-specific eligibility or permission check is failing.

All four password variables are sealed, and no manual password changes have been applied.

Please keep this request open pending human review and provide the supported path to complete credential rotation while preserving existing data, volumes, and variable references. I can provide the relevant identifiers privately.


Status changed to Awaiting Railway Response Railway • 26 days ago


Hi Bruno, the empty staged patches from October 23 that you found match an issue we're seeing in several environments since the incident, and they may be what's blocking Regenerate. Clearing them needs engineering, so we've passed your four database services to the team. We'll reply here once they're cleared and Regenerate is available. Until then, please keep avoiding manual password edits.


Status changed to Awaiting User Response Railway • 26 days ago


Status changed to Awaiting Railway Response angelo-railway • 26 days ago


angelo-railway

Hi Bruno, the empty staged patches from October 23 that you found match an issue we're seeing in several environments since the incident, and they may be what's blocking Regenerate. Clearing them needs engineering, so we've passed your four database services to the team. We'll reply here once they're cleared and Regenerate is available. Until then, please keep avoiding manual password edits.

brunolliveira
OP

25 days ago

Thank you for confirming and escalating this to engineering. We will keep the database passwords unchanged and wait for your confirmation that the empty patches have been cleared and Regenerate is available on all four services.


drgibbs11

24 days ago

For anyone else who lands here with Regenerate greyed out: the advice above is the key part. Don't edit the password variables or run ALTER USER while you wait. Sealed values can't be read back, so a mismatch is hard to repair.

Things you can still do in the meantime:

  • If the database has a public TCP proxy and nothing outside Railway needs it, remove it. Rotating the password and closing public access are two separate fixes.
  • Rotate everything else: create the new key, redeploy, check that it works, then revoke the old one.
  • Once Regenerate works again, redeploy every service that references the password or the database URL.

I wrote these steps up as a free checklist: https://github.com/mausoftskills/railway-post-breach-checklist


Welcome!

Sign in to your Railway account to join the conversation.

Loading...