Private networking dead between two services in the same project + environment
myfiredata
HOBBYOP

5 days ago

Two services in one project and environment - a Node app and a

Railway Postgres - lost private networking mid-morning and have not

recovered. Redeploying BOTH services changed nothing. Public networking on

both services is completely unaffected the whole time.

SYMPTOM

The app cannot open a TCP connection to .railway.internal:5432.

Connections TIME OUT. They are not refused.

WHAT WORKS

  • Internal DNS resolves correctly, in both directions.

  • Postgres is listening: listen_addresses='*', and /proc/net/tcp6 shows

    :::5432 in state 0A (LISTEN).

  • Inside the Postgres container, connecting to 127.0.0.1:5432 succeeds.

  • The app container's public egress is perfectly healthy: TCP to an

    external host and to 1.1.1.1:443 both connect instantly.

  • Both containers have a railnet0 interface, each with an fd12:… address,

    and both addresses sit inside the same /64.

WHAT FAILS

  • app -> db.railway.internal:5432 times out on BOTH address families —

    the fd12:… IPv6 AND the 10.x IPv4 that internal DNS returns.

  • db -> app.railway.internal:8080 ALSO times out.

    So it fails bidirectionally, not just inbound to the database.

ALREADY RULED OUT — not a listen / IPv6-binding problem

I found the earlier thread where PostgREST wasn't binding to IPv6 and the

fix was PGRST_SERVER_HOST=*6. That is not this. Postgres here is already

listening on the IPv6 wildcard (verified in /proc/net/tcp6), and crucially

my connections TIME OUT rather than being REFUSED. Refused means packets

arrived and nothing was listening; timeout means they never arrived. The

IPv4 address internal DNS hands back times out identically, so there is no

working family to fall back to — and no listen setting on either service

could make traffic fail in BOTH directions.

TIMELINE

  • Working normally all morning; last successful DB write ~2h before the

    break.

  • A deploy replaced the app container. The new container NEVER reached

    Postgres — connection timeout logged ~2 minutes after boot. The commit

    it deployed added a single unused script file: no application code, no

    dependency, no config change.

  • Redeployed the app service. Still broken.

  • Redeployed the Postgres service (it had been up 12 days). Still broken.

  • After the Postgres redeploy its private IPv6 address changed, and

    internal DNS updated to the new address correctly — but connectivity

    still did not return.

status.railway.com showed no incident and reported 100% private-networking

uptime for US West across the entire window.

QUESTIONS

  1. What can break the mesh between two containers like this, when both

    have railnet0 interfaces in the same /64 and DNS is resolving fine?

  2. Is there anything that re-establishes private networking short of

    deleting and recreating the services?

  3. Why does a total private-network failure produce no signal at all —

    no incident, no warning, deploys still reported healthy?

Solved$10 Bounty

Pinned Solution

Try to redeploy source image instead of a normal redeploy. You can do it by selecting your Postgres service, press CTRL+K, then select redeploy source image.

2 Replies

Railway
BOT

5 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 5 days ago


Try to redeploy source image instead of a normal redeploy. You can do it by selecting your Postgres service, press CTRL+K, then select redeploy source image.


myfiredata
HOBBYOP

4 days ago

Resolved — thank you, that did it!!! Woot!

"Redeploy source image" on the Postgres service restored private networking

immediately. For anyone finding this later, the key detail is that this is NOT

the same as a normal redeploy:

  • plain redeploy of the app service -> no change (tried twice)
  • plain redeploy of the Postgres service -> no change
  • REDEPLOY SOURCE IMAGE on Postgres -> fixed instantly

Roughly 22 hours of downtime, none of which plain redeploys could touch.

CLI equivalent if you prefer it to CTRL+K:

railway redeploy --service <name> --from-source

Verified after: TCP from the app container to .railway.internal:5432

connected immediately, and a real query over the private address succeeded.

Moved off the temporary public TCP proxy and deleted it.

One piece of feedback: a total private-network failure currently produces no

signal at all. No incident, the status page reported 100% private-networking

uptime for the region throughout, and deploys kept reporting healthy because

the platform only checks that the container started. I found it by accident

while looking at something unrelated. Anything that surfaced "this service

can no longer reach its database" would have saved the entire outage.


Status changed to Solved myfiredata 4 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...