We do cycle out our HTTP proxies every 24 hours, this looks to be the cause of what you are seeing.

There are only 15,000-20,000 websocket connections, so based on this graph it looks like Railway is:

1. Putting all those connections on a single proxy (or restarting all proxies at the same time)

2. On reconnect, the connections get routed to another proxy that still needs to be cycled.

3. This somehow happens 8 times in a row.

Even if Railway cycles HTTP proxies every 24 hours (which is perfectly reasonable), I still find the repeated spikes unusual and concerning. It would also be appreciated if the load was spread in a way that not all sockets are lost at the same time.

If this could be surfaced to the networking team it'd be greatly appreciated.

We have eight proxies in the US-West2 region, and when we restart them, they are all staggered one after the other.

All the open connections to your application would be spread out among those 8 proxies.

And that is what these spikes are, each individual proxy restart.

I've raised this to Infra for additional comments.

Heard back from the networking team:

On reconnect, the connections get routed to another proxy that still needs to be cycled.

This is one of the main issues. We also aren't draining the connections as gracefully as we should be. When draining, we drop all connections simultaneously, instead of allowing gradual connection closures with H2 GOAWAY.

So it turns out that our networking team was already painfully aware of the issue, but they are all tied up in projects they had already started and committed to, and those need to be completed before tackling this issue, so for right now, there isn't anything we can do short term, but this will get improvements down the line.

Thanks, I really appreciate the update. It's very unfortunate behavior but I'm glad it's on the radar, and now that I know the cause I can look into how to mitigate it at the app level too :)

You aren't the only one reporting WebSocket connection closures either, so this will likely be next up after we bring IPv4 support to the private network.