[URGENT PROD DOWN LIVE] Networking / Outbound Connection Problems (reopened)
alfredbenoel
PROOP

2 months ago

Hi railway team,

We just faced this issue (as documented on https://station.railway.com/questions/networking-outbound-connection-problem-caafa330)

Here is the deployment hash where it happened: 0bb948af

We managed to recover by manually re-deploying the pod again.

$20 Bounty

9 Replies

Status changed to Awaiting Railway Response Railway 2 months ago


alfredbenoel
PROOP

2 months ago

first occurrence of the 502 connectivity error happened at Jun 20, 2026, 10:29:49 PM GMT+8.


alfredbenoel
PROOP

2 months ago

This has been happening constantly (almost every weekend for the past 2 months at this point) and we haven't received any of much support, which is pretty frustrating :(

If there is some inherent unknown issue with the internal mesh that the team needs live data to investigate, i get it, but having to manually redeploy with no programmatic way to monitor these connectivity issues and automatically trigger a redeploy in railway is what makes this even more frustrating. Is there a way that we could possibly expose the outbound connectivity issues with the health check and trigger a redeploy of an existing pod?


alfredbenoel
PROOP

2 months ago

This happened again to us (Jun 22, 2026, 2:41:03 PM GMT+8, deployment hash: ccc24231)

We re-did a review of our changes and have confirmed with 100% certainty that this isn't some code-side issue.


Thanks for the persistence and the deploy hashes, and sorry this keeps eating your weekends. Confirmed: this is the intermittent outbound-connectivity drop we're actively investigating, the host your pod is on loses outbound while internal networking and DNS keep working, which is why a redeploy (moving you to a fresh host) restores it. Your service is on the default shared egress in our Singapore zone (asia-southeast1 / eqsg3a); it isn't a code issue on your side.

On your actual ask, automating the recovery. Our built-in healthcheck is inbound-only, so it won't catch an outbound-only drop, and a crash-restart would just land you back on the same host. The reliable approach is a small watchdog in your service: periodically probe your critical outbound dependency and, after N consecutive failures, call the Railway API to redeploy yourself (which moves you to a fresh host):

mutation { serviceInstanceRedeploy(serviceId: "97d0af4c-4abe-4966-9298-add279e3e335", environmentId: "f4fdc553-a705-4004-8830-03b7ac53801a") }

(use a Railway API token; gate it behind several consecutive failures so it can't loop). That turns the manual redeploy into an automatic one until we land the platform fix.

Separately, moving this service to HA static egress would route your outbound through dedicated gateway IPs (208.77.246.240-242) instead of the shared per-host NAT, which should sidestep the per-host drop entirely. It changes your outbound IPs, so you'd re-allowlist them with any vendors first. Want me to enable that for you?

And next time it's actively happening, drop the timestamp here and we'll pull live data while it's broken, that's what most helps us nail the root cause. — Angelo


Status changed to Awaiting User Response Railway 2 months ago


alfredbenoel
PROOP

2 months ago

Thank you so much for this angelo and yes would it be possible to route our outbound through a dedicated gateway IP instead.


Status changed to Awaiting Railway Response Railway 2 months ago


You can enable this yourself right now. Go to your service's Settings tab, scroll to the Networking section, and toggle "Enable Static IPs." You'll be shown three dedicated outbound IPv4 addresses for your region. The new IPs take effect after the next deploy. Make sure to update any downstream allowlists with the new IPs before deploying.


Status changed to Awaiting User Response Railway 2 months ago


alfredbenoel
PROOP

2 months ago

Hey railway team/angelo, while moving to static ip address solved out 502 errors, we've been experiencing intermittent 499 errors on 2 separate occasions, that was fixed by re-deploying the pod.

here are the 2 commit hashes + timestamp:

679fb8f9 from Jun 25, 2026, 2:07:13 AM GMT+8

and

9bd3be8f from Jun 24, 2026, 9:04:32 AM GMT+8

The only reason i can think of is that i hadn't enabled static ip address to the frontend pod, because i assumed that services in a project are within the same network mesh, but am i incorrect to assume this and instead using a shared pooler even for services within the same project is the correct way to do this?


Status changed to Awaiting Railway Response Railway about 2 months ago


Railway
BOT

2 months ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 2 months ago


alfredbenoel
PROOP

2 months ago

(editing to add more details)

Hi, is happening live to us right now, we have re-deployed 3 times, and it doesn't seem to be able to recover anymore.

It always seem to starts off with normal traffic flow -> 499 (Jun 30, 2026, 12:49:29 PM GMT+8, hash: 97503f6e) -> 502

This same pattern has been observed


alfredbenoel
PROOP

2 months ago

@angelo-railway this is happening live


Welcome!

Sign in to your Railway account to join the conversation.

Loading...