a month ago
We are experiencing a critical issue with internal service-to-service communication in our production environment. Identical configuration works flawlessly in our acceptance environment, but fails in production.
Environment Details:
Project ID: a7e5849c-cbde-48f7-a765-fd62d670758b
Production Environment ID: 665f0795-45b5-4e00-a9b0-f67545b41232
Acceptance Environment ID: e2d00610-2b29-40db-9983-753e53ddb627 (working)
Problem:
Internal service-to-service communication fails with IPv6 connection refused errors. The Tracking proxy (Caddy reverse proxy) cannot reach Rybbit Client and Rybbit Backend services via their internal Railway DNS names.
Error Log:
{ "message": "dial tcp [fd12:28b7:3311:0:1000:1a:7cae:56cc]:3002: connect: connection refused", "severity": "error", "attributes": { "err_trace": "reverseproxy.statusError (reverseproxy.go:1626)", "logger": "http.log.error" }, "status": 502}
Services Involved:
Tracking proxy (Caddy 2.11.4) → attempting to reach:
backend-ws3y.railway.internal:3001 (Rybbit Backend)
client-q3wd.railway.internal:3002 (Rybbit Client)
Key Observations:
All services are running and healthy (status: online)
Configuration is identical between acceptance and production environments
Internal routing works perfectly in acceptance environment
Production environment shows IPv6 connection refused errors
Services cannot communicate internally despite being in the same private network
This affects all internal service-to-service communication (tested with both Caddy proxy and simple Hono scripts)
Expected Behavior:
Services should be able to communicate via internal Railway DNS names within the same environment's private network, regardless of IPv4/IPv6 protocol.
Actual Behavior:
IPv6 connections to internal service DNS names are refused in production, while the same setup works in acceptance.
Impact:
Complete service-to-service communication failure in production. The application is non-functional.
Pinned Solution
a month ago
If your production environment was created before October 16th, 2025, it's considered a legacy environment (Only supports IPv6 inside the private network). Now, a similar issue was reported by a user a while back, they had failed connections inside the private network over IPv6 in a legacy environment. Although the docs state that binding your app to :: should not be an issue.
If your environment is a legacy one, I recommend you create a new one and migrate your production services to it, or make acceptance the new production. Since your acceptance environment is identecal and has no issues this is the most likely reason.
You can read more about this here: https://docs.railway.com/private-networking#legacy-environments
4 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 27 days ago
a month ago
Yes, multiple times. Yesterday I tried just to route the traffic with a simple script to a different service without any luck. There is just something wrong with the service-to-service communication for this specific enviroment.
a month ago
If your production environment was created before October 16th, 2025, it's considered a legacy environment (Only supports IPv6 inside the private network). Now, a similar issue was reported by a user a while back, they had failed connections inside the private network over IPv6 in a legacy environment. Although the docs state that binding your app to :: should not be an issue.
If your environment is a legacy one, I recommend you create a new one and migrate your production services to it, or make acceptance the new production. Since your acceptance environment is identecal and has no issues this is the most likely reason.
You can read more about this here: https://docs.railway.com/private-networking#legacy-environments
a month ago
Ok. If there is no other way, i'll migrate my services. But, please Railway, show some kind of warning to notify you're using a legacy environment..
Status changed to Solved 0x5b62656e5d • 25 days ago