Pre-deploy command fails Postgres authentication while main app authenticates successfully (identical credentials)
michaelejfranke-cmyk
PROOP

5 days ago

Hi Railway team,

I'm running into a confusing issue where my pre-deploy command fails

to authenticate against my Postgres service, while my main application

container in the same service authenticates successfully using the same

DATABASE_URL reference.

PROJECT CONTEXT

Project: wisdomhub-production

Environment: production

Service experiencing the issue: lps-pro (FastAPI backend)

Database service: Postgres

Plan: Pro

WHAT I'M TRYING TO DO

Configure a pre-deploy command to run Alembic migrations automatically

before each deploy:

alembic upgrade head

WHAT HAPPENS

The pre-deploy command consistently fails with:

psycopg2.OperationalError: connection to server at

"postgres.railway.internal" (fd12:904:ba0f:1:5000:29:8dfc:e7ba),

port 5432 failed: FATAL: password authentication failed for user

"postgres"

However, my main app container running in the same service connects

successfully to the same Postgres on every deploy. The app is currently

serving traffic without issues.

DIAGNOSTIC EVIDENCE

I replaced the pre-deploy command with diagnostic Python scripts that

inspect the DATABASE_URL environment variable. Here's what I found:

  1. Both contexts resolve postgres.railway.internal to the same addresses:

    • IPv6: fd12:904:ba0f:1:5000:29:8dfc:e7ba
    • IPv4: 10.252.231.186

  2. Both contexts see identical DATABASE_URL components:

    • HOST: postgres.railway.internal
    • PORT: 5432
    • USER: postgres
    • DB: railway
    • Password length: 43 chars
    • Password first 3 chars: VJK
    • Password last 3 chars: tyU
    • MD5(password)[0:8]: c16d5e3e (identical in pre-deploy and main app)

  3. I attempted direct psycopg2.connect() in the pre-deploy command,

    bypassing SQLAlchemy/Alembic, using both the hostname and the literal

    IPv4 address — both fail with the same authentication error.

  4. I tried both reference variable (${{Postgres.DATABASE_URL}}) and a

    literal static URL in the lps-pro DATABASE_URL — both fail in

    pre-deploy.

  5. The diagnostic prints succeed when the command terminates with

    exit code 0 (no actual DB connection attempted), so the pre-deploy

    container has Python, has psycopg2 installed, and has DATABASE_URL

    injected correctly. The issue is specifically the Postgres

    authentication.

CONCLUSION

With identical credentials (verified by hash), identical resolved IPs,

and identical environment variables, the main app authenticates but the

pre-deploy container does not. This appears to be a platform-level issue

where pre-deploy containers are being treated differently by the Postgres

service authentication layer.

Could you investigate why pre-deploy containers receive authentication

failures when main app containers with identical credentials succeed

against the same Postgres instance?

For now I have removed the pre-deploy command to keep deploys working.

I'm running Alembic migrations manually before each deploy as a

workaround.

Thank you for your help.

Best regards,

Michaele Franke

michaele.j.franke@gmail.com

Solved$20 Bounty

Pinned Solution

darseen
HOBBYTop 1% Contributor

5 days ago

You can regenerate your Postgres password in service settings > database > config. Make sure to redeploy any services that connect to the database after regenerating the password.

image.png

Attachments

image.png

4 Replies

Railway
BOT

5 days ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open Railway • 5 days ago

darseen
HOBBYTop 1% Contributor

5 days ago

Two workarounds I can suggest:

  1. Use the public URL to connect to Postgres in the pre-deploy command.
  2. Move the database migration to the start command before running your app.

I believe the second option is another valid way of running database migrations, if you're facing problems with the pre-deploy command.

michaelejfranke-cmyk
PROOP

5 days ago

URGENT UPDATE — Bug now affects production runtime, not just pre-deploy.

Status: Production backend on Railway cannot connect to attached Postgres service. Same FATAL: password authentication failed for user "postgres" error as originally reported, now blocking all DB-dependent endpoints (login, etc.). Backend service is "Online" in dashboard but every connection attempt fails auth.

Diagnostic performed:

  1. Verified DATABASE_URL on backend service matches DATABASE_URL on Postgres service (same user, same password, same host postgres.railway.internal:5432).
  2. Restarted Postgres service via Deployments → Restart Successful. Bug persists post-restart.
  3. Tested external connection via psql 17.9 using DATABASE_PUBLIC_URL. Same FATAL auth error from external network — confirming the issue is inside the Postgres cluster (pg_authid desynchronized from exposed POSTGRES_PASSWORD variable), not a network issue.
  4. No infrastructure incident reported on status.railway.com (Fully Operational).
  5. No notifications from Railway support indicating known issue.

Conclusion: pg_authid inside the Postgres container is out of sync with the POSTGRES_PASSWORD variable Railway exposes. Standard customer-side fixes are blocked because:

  • Railway docs confirm POSTGRES_PASSWORD variable only sets initial password, not subsequent ones.
  • Manual ALTER USER via psql requires authenticating first, which is exactly what fails.
  • Restart did not resolve.

Request to Railway team:

  1. Please intervene directly on the Postgres cluster of this project to reset the postgres user password inside pg_authid.
  2. Please rotate to a new password value (do not reuse current) — current password value was inadvertently exposed in an external debugging session and should be considered compromised.
  3. After reset, please confirm the new password matches the DATABASE_URL exposed in the dashboard so we can update our backend service variable accordingly.

Impact: Production release blocked. Backup pre-incident exists (2026-05-17 17:10 UTC, 119 MB) as safety net but restore would lose recent schema migrations 0013→0016.

Priority requested: Urgent / Production Down.

darseen
HOBBYTop 1% Contributor

5 days ago

You can regenerate your Postgres password in service settings > database > config. Make sure to redeploy any services that connect to the database after regenerating the password.

image.png

Attachments

image.png

darseen

You can regenerate your Postgres password in **service settings > database > config.** Make sure to redeploy any services that connect to the database after regenerating the password. ![image.png](https://station-server.railway.com/attachments/att_01krvzsz9efk2t2cvm5h6ree47)

michaelejfranke-cmyk
PROOP

5 days ago

darseen — confirmed working. The Regenerate Password button in Database > Config was exactly what was needed. After clicking it and updating DATABASE_URL on the backend service manually (since it was set as a literal, not a reference variable), the backend reconnected immediately and authentication now works end-to-end.

For anyone hitting this in the future: the POSTGRES_PASSWORD service variable does NOT trigger an ALTER USER inside pg_authid. The dedicated Regenerate button does. These are completely different operations even though they appear similar in the UI.

Thanks again — saved us hours of waiting for support intervention.

+11

Status changed to Solved 0x5b62656e5d • 5 days ago

Welcome!

Sign in to your Railway account to join the conversation.

Login
Loading...