5 days ago
Two services in one project and environment - a Node app and a
Railway Postgres - lost private networking mid-morning and have not
recovered. Redeploying BOTH services changed nothing. Public networking on
both services is completely unaffected the whole time.
SYMPTOM
The app cannot open a TCP connection to .railway.internal:5432.
Connections TIME OUT. They are not refused.
WHAT WORKS
-
Internal DNS resolves correctly, in both directions.
-
Postgres is listening: listen_addresses='*', and /proc/net/tcp6 shows
:::5432 in state 0A (LISTEN).
-
Inside the Postgres container, connecting to 127.0.0.1:5432 succeeds.
-
The app container's public egress is perfectly healthy: TCP to an
external host and to 1.1.1.1:443 both connect instantly.
-
Both containers have a railnet0 interface, each with an fd12:… address,
and both addresses sit inside the same /64.
WHAT FAILS
-
app -> db.railway.internal:5432 times out on BOTH address families —
the fd12:… IPv6 AND the 10.x IPv4 that internal DNS returns.
-
db -> app.railway.internal:8080 ALSO times out.
So it fails bidirectionally, not just inbound to the database.
ALREADY RULED OUT — not a listen / IPv6-binding problem
I found the earlier thread where PostgREST wasn't binding to IPv6 and the
fix was PGRST_SERVER_HOST=*6. That is not this. Postgres here is already
listening on the IPv6 wildcard (verified in /proc/net/tcp6), and crucially
my connections TIME OUT rather than being REFUSED. Refused means packets
arrived and nothing was listening; timeout means they never arrived. The
IPv4 address internal DNS hands back times out identically, so there is no
working family to fall back to — and no listen setting on either service
could make traffic fail in BOTH directions.
TIMELINE
-
Working normally all morning; last successful DB write ~2h before the
break.
-
A deploy replaced the app container. The new container NEVER reached
Postgres — connection timeout logged ~2 minutes after boot. The commit
it deployed added a single unused script file: no application code, no
dependency, no config change.
-
Redeployed the app service. Still broken.
-
Redeployed the Postgres service (it had been up 12 days). Still broken.
-
After the Postgres redeploy its private IPv6 address changed, and
internal DNS updated to the new address correctly — but connectivity
still did not return.
status.railway.com showed no incident and reported 100% private-networking
uptime for US West across the entire window.
QUESTIONS
-
What can break the mesh between two containers like this, when both
have railnet0 interfaces in the same /64 and DNS is resolving fine?
-
Is there anything that re-establishes private networking short of
deleting and recreating the services?
-
Why does a total private-network failure produce no signal at all —
no incident, no warning, deploys still reported healthy?
Pinned Solution
5 days ago
Try to redeploy source image instead of a normal redeploy. You can do it by selecting your Postgres service, press CTRL+K, then select redeploy source image.
2 Replies
5 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 5 days ago
5 days ago
Try to redeploy source image instead of a normal redeploy. You can do it by selecting your Postgres service, press CTRL+K, then select redeploy source image.
4 days ago
Resolved — thank you, that did it!!! Woot!
"Redeploy source image" on the Postgres service restored private networking
immediately. For anyone finding this later, the key detail is that this is NOT
the same as a normal redeploy:
- plain redeploy of the app service -> no change (tried twice)
- plain redeploy of the Postgres service -> no change
- REDEPLOY SOURCE IMAGE on Postgres -> fixed instantly
Roughly 22 hours of downtime, none of which plain redeploys could touch.
CLI equivalent if you prefer it to CTRL+K:
railway redeploy --service <name> --from-sourceVerified after: TCP from the app container to .railway.internal:5432
connected immediately, and a real query over the private address succeeded.
Moved off the temporary public TCP proxy and deleted it.
One piece of feedback: a total private-network failure currently produces no
signal at all. No incident, the status page reported 100% private-networking
uptime for the region throughout, and deploys kept reporting healthy because
the platform only checks that the container started. I found it by accident
while looking at something unrelated. Anything that surfaced "this service
can no longer reach its database" would have saved the entire outage.
Status changed to Solved myfiredata • 4 days ago