Our healthchecks started failing off hours (no deploys from our team). Digging deeper we discovered three of our services were no longer able to reach one of the other services. Investigating, we saw Railway initiated a redeploy of that service that was no longer reachable. This was the same issue we encountered last week during the major incident/outage.

To resolve, we redeployed the unreachable service, and all three of the other services that were trying to talk to it (restarting did not work).

Some additional debugging information:

• generating an external domain for the unreachable service worked - we were able to use the external domain to talk to our service

• internal network communications were timing out - not "connection refused"

This makes me think there is some DNS action that isn't running when Railway initiates a redeploy. Is the internal domain still pointing to the old/spun down container?

Either way, for other folks that might be experiencing this the tldr; resolution is:

• redeploy the services that are trying to talk to the service that Railway initiated the deploy on