Internal Variable References Completely Broken — Production Down

14 days ago

We are experiencing a critical production outage caused by Railway's internal variable references (${{ServiceName.VAR}}) failing to resolve between services in the same environment.

Our services are configured to communicate using Railway's own recommended variable reference syntax, such as:

  • ${{PgBouncer.RAILWAY_PRIVATE_DOMAIN}} — referenced in the API service to reach PgBouncer
  • ${{DB-API-PRD.PGPASSWORD}} — referenced in PgBouncer to authenticate against PostgreSQL
  • ${{DB-API-PRD.PGUSER}} — same as above
  • ${{DB-API-PRD.RAILWAY_PRIVATE_DOMAIN}} — referenced in PgBouncer to reach the database host

None of these references are resolving. The variables are being passed as literal strings or empty values to the containers, causing our entire infrastructure to fail at startup.

This is not a misconfiguration on our end. These are standard Railway variable references following the exact syntax documented by Railway. The services are in the same project and environment. There were no changes to the

service names or variable references.

This issue started after a reported Railway infrastructure incident that forced us to redeploy our services. Since then, internal variable references have simply stopped working — something that was functioning correctly

before and that we have no control over.

The impact is severe: our production environment is completely down. We had to resort to hardcoding sensitive credentials directly into environment variables as a workaround, which is a security concern and something we should

never have to do on a platform like Railway.

We expect internal variable references to be a core, reliable feature of Railway. Having production go down because of a platform-level failure in variable resolution is unacceptable. We need this investigated and fixed

urgently.

Environment details:

  • Project: Golf Billions — User API PRD
  • Services affected: API, PgBouncer, DB-API-PRD
  • All services are in the same Railway environment
$20 Bounty

5 Replies

Status changed to Open Railway 14 days ago


14 days ago

These variables were working correctly before the Railway incident caused the problem with Google Cloud. I restarted PgBouncer, and the DB-API-PRD database, after the Railway incident, is no longer working. To work around the situation, I inserted the data into the variables in a fixed way and it worked. But the variables still don't work.


14 days ago

If I share my screen with variables, I will be exposing sensitive data.


14 days ago

Yes, they are.


Have you tried to select reference variables from the dropdown list, instead of manually typing them?

image.png

Attachments


marcelo2212

These variables were working correctly before the Railway incident caused the problem with Google Cloud. I restarted PgBouncer, and the DB-API-PRD database, after the Railway incident, is no longer working. To work around the situation, I inserted the data into the variables in a fixed way and it worked. But the variables still don't work.

Also , it's recommended to redeploy your affected services from the incident using CTRL + K then selecting redeploy source image, or deploy latest commit.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...