14 days ago
We are experiencing a critical production outage caused by Railway's internal variable references (${{ServiceName.VAR}}) failing to resolve between services in the same environment.
Our services are configured to communicate using Railway's own recommended variable reference syntax, such as:
- ${{PgBouncer.RAILWAY_PRIVATE_DOMAIN}} — referenced in the API service to reach PgBouncer
- ${{DB-API-PRD.PGPASSWORD}} — referenced in PgBouncer to authenticate against PostgreSQL
- ${{DB-API-PRD.PGUSER}} — same as above
- ${{DB-API-PRD.RAILWAY_PRIVATE_DOMAIN}} — referenced in PgBouncer to reach the database host
None of these references are resolving. The variables are being passed as literal strings or empty values to the containers, causing our entire infrastructure to fail at startup.
This is not a misconfiguration on our end. These are standard Railway variable references following the exact syntax documented by Railway. The services are in the same project and environment. There were no changes to the
service names or variable references.
This issue started after a reported Railway infrastructure incident that forced us to redeploy our services. Since then, internal variable references have simply stopped working — something that was functioning correctly
before and that we have no control over.
The impact is severe: our production environment is completely down. We had to resort to hardcoding sensitive credentials directly into environment variables as a workaround, which is a security concern and something we should
never have to do on a platform like Railway.
We expect internal variable references to be a core, reliable feature of Railway. Having production go down because of a platform-level failure in variable resolution is unacceptable. We need this investigated and fixed
urgently.
Environment details:
- Project: Golf Billions — User API PRD
- Services affected: API, PgBouncer, DB-API-PRD
- All services are in the same Railway environment
5 Replies
Status changed to Open Railway • 14 days ago
14 days ago
These variables were working correctly before the Railway incident caused the problem with Google Cloud. I restarted PgBouncer, and the DB-API-PRD database, after the Railway incident, is no longer working. To work around the situation, I inserted the data into the variables in a fixed way and it worked. But the variables still don't work.
14 days ago
Have you tried to select reference variables from the dropdown list, instead of manually typing them?
Attachments
marcelo2212
These variables were working correctly before the Railway incident caused the problem with Google Cloud. I restarted PgBouncer, and the DB-API-PRD database, after the Railway incident, is no longer working. To work around the situation, I inserted the data into the variables in a fixed way and it worked. But the variables still don't work.
14 days ago
Also , it's recommended to redeploy your affected services from the incident using CTRL + K then selecting redeploy source image, or deploy latest commit.