11 days ago
Evidence:
nginx (frontend) repeatedly fails to reach backend's private domain: no live upstreams while connecting to upstream and upstream timed out (110: Operation timed out).
The failing upstream resolves inconsistently — sometimes as hostname game-intelligence.railway.internal, sometimes as raw IPv6 literal [fd12:b536:1bd3:1:2000:54:6c72:d11d]:8080, for the same endpoint seconds apart.
Backend service (backend) shows zero incoming request logs during these failure windows — requests never arrive server-side, confirming this isn't an app-level issue.
Backend Metrics show idle CPU (~0.05 vCPU) and ~100MB memory, well under the 1GB/2vCPU limit — not a resource/OOM issue.
Both services are in the same region (US West), ruling out cross-region networking.
Issue persists after a full redeploy of the frontend service (new deployment 6cad2fcb-097b-..., still active as of this report).
Ask: Please investigate private network/DNS resolution for these two services — the IPv6 address flapping suggests an internal service-discovery issue, not application code.
1 Replies
11 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 11 days ago
11 days ago
The evidence does show this is below the backend application handler, but the IPv6 literal alone is not enough to prove Railway DNS flapping. Railway private networking uses internal DNS and IPv6; in legacy environments it is IPv6-only. A raw fd12:* address in Nginx logs is expected for Railway private networking.
I would check two things first:
- Backend bind address
Railway recommends services using private networking listen on :: / [::], not only 127.0.0.1 or 0.0.0.0.
Inside the backend container, verify:
ss -lntp | grep 8080
Healthy for private IPv6 should look like:
If it only listens on 127.0.0.1:8080 or 0.0.0.0:8080, Nginx can resolve the private IPv6 but the app may never receive the request.
- Nginx DNS/runtime resolution
If Nginx config uses a static upstream like:
upstream backend {
server game-intelligence.railway.internal:8080;}
Nginx may resolve that name at startup and keep a stale address after backend redeploys/private IP changes. That can produce no live upstreams and timeouts while backend logs show zero requests.
Use runtime DNS resolution with Railway’s container resolver, for example by templating /etc/resolv.conf into Nginx and using variable proxy_pass:
resolver <nameserver-from-/etc/resolv.conf> valid=10s ipv6=on;
set $backend "http://game-intelligence.railway.internal:8080";
proxy_pass $backend;
Also run this from the frontend container during a failure window:
getent ahosts game-intelligence.railway.internal
curl -v -g -6 --connect-timeout 3 http://game-intelligence.railway.internal:8080/health
If getent resolves but curl -6 times out and backend is listening on [::]:8080, then it is strong evidence for a Railway private-network path issue. If backend is not listening on [::]:8080, fix the backend bind first.