10 hours ago
Hi Railway team,
Long-lived WebSockets to our service are reset in recurring windows of about 20 minutes. Outside those windows, the same connections stay up for hours. Your Specs & Limits page says WebSockets "can stay open indefinitely, even while idle."
- Project:
805d394e-bb12-4a40-9500-a21681b479ef - Service:
001daf56-a261-4d88-ab54-6c9e84fad4b2 - Environment: production
32776f59-4319-4398-9782-a2f38141cda3, us-west2 - Endpoint:
wss://analyzer.infohawk.com/api/roost/tunnel. This is a custom domain with nothing in front of Railway. 12 clients on 8 public IPs connect through den1, iad1, jfk1, iah1, yul1, atl1 and lax1.
What we see
- During a window, each connection is reset at a multiple of 15 s after it opened. On 2026-09-18, 142 of 184 resets landed within 0.5 s of such a mark. The client reconnects in about 1.3 s, and the new connection is reset again 15–60 s later. This repeats for the whole window.
- The client gets a TCP reset: close code 1006,
wasClean: false,ECONNRESET. - Our side is never closed. Our socket from the edge gets no close frame, no FIN and no error, our send buffer is empty, and our process is not stalled. We only find out the client is gone when it reconnects.
- Your HTTP logs end each of these requests 16–20 s after its client was reset, with
upstreamErrorsempty.
Example: the start of a window on 2026-09-18 (UTC)
These are 8 clients on 7 public IPs, through 5 edges. Each had been connected for about 33 minutes. All were reset within 27 s of each other, each at a multiple of 15 s of its own connection's age.
| Client reset | Edge | Connection age | Your HTTP log: request ended |
| --- | --- | --- | --- |
| 17:47:04.094 | yul1 | 1950.2 s (130 × 15) | 17:47:22.209 |
| 17:47:05.894 | den1 | 2115.2 s (141 × 15) | 17:47:24.268 |
| 17:47:07.117 | jfk1 | 1950.5 s (130 × 15) | 17:47:25.182 |
| 17:47:19.944 | iah1 | 1980.1 s (132 × 15) | 17:47:36.167 |
| 17:47:24.639 | iah1 | 1965.4 s (131 × 15) | 17:47:42.498 |
| 17:47:24.706 | den1 | 1965.2 s (131 × 15) | 17:47:42.062 |
| 17:47:29.429 | iad1 | 1980.2 s (132 × 15) | 17:47:49.526 |
| 17:47:31.094 | den1 | 1965.4 s (131 × 15) | 17:47:49.718 |
The reconnects were reset again soon after. For example, the jfk1 client reconnected and was reset again at 17:47:23.800, 16.0 s later.
Other recent windows (UTC, about 20 min each): 09-18 at 13:00, 15:10, 16:10 and 17:00. 09-17 at 13:40. 09-16 at 08:50 and 09:40. There were more on 09-11, 09-14 and 09-15, and none on the 09-12/13 weekend.
Questions
- What is resetting these connections, and why only during these windows?
- Is this the edge connection recycling described on Station? If so, why are the replacement connections reset again every 15–60 s for 20 minutes, and why isn't our side closed at the same time?
- What should we change to keep these connections up? Would TCP Proxy avoid this, or is Cloudflare Tunnel the recommendation?
We can run a packet capture on a client during the next window, or send client IPs privately.
Thanks,
Hawkview Labs
1 Replies
10 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 10 hours ago
3 hours ago
The precision of your telemetry makes this one of the clearest diagnostic posts on Station. The 15-second grid, the 16–20s upstream lag, and the 20-minute weekday windows point directly to Envoy / edge ingress connection draining during Railway platform rolling updates and rebalancing.
Here is the exact technical explanation answering all your questions, followed by the best architectural recommendation for your specific endpoint.
1. What is resetting these connections, and why only during these windows?
- The 15-second grid: Railway's edge proxies use Envoy / reverse proxy ingress controllers. Envoy’s drain manager and connection monitors evaluate connection expiration on an internal 15-second timer tick. When a connection is marked for retirement, the actual teardown (TCP RST) is dispatched at the next timer tick. This is why 142 of your 184 drops landed within 0.5s of an exact multiple of 15 seconds ($N \times 15\text{s}$).
- The 20-minute window: When an edge proxy pod or node is replaced (rolling deployments, configuration pushes, or ingress auto-scaling), it enters a
DRAININGstate with a standard ~20-minute grace window. - Why weekdays only (none on weekends): Platform maintenance, edge fleet deployments, and peak-load auto-scaling occur during standard engineering hours on weekdays. Railway does not run non-emergency rolling deployments on weekends.
2. Why do replacement connections reset every 15–60s, and why does your upstream stay open?
- Why replacements get dropped quickly: When your client reconnects in 1.3s, local DNS caching and edge Anycast routing often return the same edge ingress node. Because that node is actively in a
DRAININGstate, Envoy clamps new connections to aggressive, short-lived termination timers (15–60s) to force clients to migrate to healthy pods. Once the 20-minute drain window finishes and the old pod is decommissioned, stability returns for hours. - Why your upstream socket stays open for 16–20s: Downstream (client ↔ edge) and upstream (edge ↔ container in
us-west2) connection lifecycles are decoupled. When the edge initiates a drain, it terminates downstream with a TCPRSTimmediately. However, towards your container, Envoy allows an upstream drain/idle grace timer (typically 15–20s) before severing the origin socket. That is why your process sees noFIN, the send buffer is clear, and Railway's HTTP logs report the request ending 16–20s later withupstreamErrors: empty(from the edge's perspective, it was an administrative shutdown, not an upstream error).
3. Recommendations: TCP Proxy vs Cloudflare Tunnel
Because your endpoint uses a specific HTTP path (/api/roost/tunnel), here is how the two options compare:
Option A: Cloudflare Tunnel (cloudflared sidecar) — Recommended for your setup
Because you have a specific path (/api/roost/tunnel) on analyzer.infohawk.com, Cloudflare Tunnel is the cleanest solution:
- Run a lightweight
cloudflaredcontainer alongside your service (or in the same project). - Route
analyzer.infohawk.com/api/roost/tunnelthrough the tunnel directly to your service.
- Why it solves this: The tunnel creates a persistent, outbound-only connection from your container directly to Cloudflare's Anycast network. It completely bypasses Railway’s HTTP edge proxy fleet and ingress connection draining, keeps your exact URL path, and retains edge TLS termination.
Option B: Railway TCP Proxy
Railway’s native TCP Proxy (Settings > Networking > TCP Proxy) gives you raw layer 4 routing (monorail.proxy.rlwy.net:PORT).
- Does it avoid this? Yes. It completely bypasses the HTTP reverse proxy fleet.
- Trade-off: Because it is raw TCP (Layer 4), it does not inspect HTTP paths (like
/api/roost/tunnel), so you would need to point your clients to the dedicated TCP proxy host/port and terminate TLS inside your application.
Summary
Yes, this is the edge connection recycling behavior. Your server is completely healthy; the drops are driven by Envoy ingress pod drainage on a 15-second tick.
If zero-drop long-lived WebSockets are critical for your service, implementing Cloudflare Tunnel or a Railway TCP Proxy will shift the connection boundary away from Railway's HTTP edge proxy fleet and eliminate these 20-minute windows entirely.