Internal Variable References Completely Broken — Production Down

2 hours ago

We are experiencing a critical production outage caused by Railway's internal variable references (${{ServiceName.VAR}}) failing to resolve between services in the same environment.

Our services are configured to communicate using Railway's own recommended variable reference syntax, such as:

  • ${{PgBouncer.RAILWAY_PRIVATE_DOMAIN}} — referenced in the API service to reach PgBouncer
  • ${{DB-API-PRD.PGPASSWORD}} — referenced in PgBouncer to authenticate against PostgreSQL
  • ${{DB-API-PRD.PGUSER}} — same as above
  • ${{DB-API-PRD.RAILWAY_PRIVATE_DOMAIN}} — referenced in PgBouncer to reach the database host

None of these references are resolving. The variables are being passed as literal strings or empty values to the containers, causing our entire infrastructure to fail at startup.

This is not a misconfiguration on our end. These are standard Railway variable references following the exact syntax documented by Railway. The services are in the same project and environment. There were no changes to the

service names or variable references.

This issue started after a reported Railway infrastructure incident that forced us to redeploy our services. Since then, internal variable references have simply stopped working — something that was functioning correctly

before and that we have no control over.

The impact is severe: our production environment is completely down. We had to resort to hardcoding sensitive credentials directly into environment variables as a workaround, which is a security concern and something we should

never have to do on a platform like Railway.

We expect internal variable references to be a core, reliable feature of Railway. Having production go down because of a platform-level failure in variable resolution is unacceptable. We need this investigated and fixed

urgently.

Environment details:

  • Project: Golf Billions — User API PRD
  • Services affected: API, PgBouncer, DB-API-PRD
  • All services are in the same Railway environment
$20 Bounty

2 Replies

Status changed to Open Railway about 2 hours ago


asepsaputra
HOBBYTop 5% Contributor

27 minutes ago

Hi,

Can you please share screenshots of the Variables tab for the related services?

Specifically:

  1. API service variables

    To verify the reference to:

${{PgBouncer.RAILWAY_PRIVATE_DOMAIN}}
  1. PgBouncer service variables

    To verify the references to:

${{DB-API-PRD.PGPASSWORD}}
${{DB-API-PRD.PGUSER}}
${{DB-API-PRD.RAILWAY_PRIVATE_DOMAIN}}

Please hide or blur any sensitive values, but keep the variable names and reference syntax visible.

It would also help to see whether Railway shows the references as resolved values or still as literal ${{...}} strings, the concept of using RAILWAY_PRIVATE_DOMAIN for service-to-service communication is correct, but we need to confirm the references are pointing to the correct service names, variables, and environment.


asepsaputra
HOBBYTop 5% Contributor

25 minutes ago

Just to confirm, are the services actually named DB-API-PRD and PgBouncer in Railway? Variable references depend on the exact service name, so any typo or name mismatch could cause them not to resolve.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...