18 days ago
Hi Railway team,
Long-lived WebSockets to our service are reset in recurring windows of about 20 minutes. Outside those windows, the same connections stay up for hours. Your Specs & Limits page says WebSockets "can stay open indefinitely, even while idle."
- Project:
805d394e-bb12-4a40-9500-a21681b479ef - Service:
001daf56-a261-4d88-ab54-6c9e84fad4b2 - Environment: production
32776f59-4319-4398-9782-a2f38141cda3, us-west2 - Endpoint:
wss://analyzer.infohawk.com/api/roost/tunnel. This is a custom domain with nothing in front of Railway. 12 clients on 8 public IPs connect through den1, iad1, jfk1, iah1, yul1, atl1 and lax1.
What we see
- During a window, each connection is reset at a multiple of 15 s after it opened. On 2026-09-18, 142 of 184 resets landed within 0.5 s of such a mark. The client reconnects in about 1.3 s, and the new connection is reset again 15–60 s later. This repeats for the whole window.
- The client gets a TCP reset: close code 1006,
wasClean: false,ECONNRESET. - Our side is never closed. Our socket from the edge gets no close frame, no FIN and no error, our send buffer is empty, and our process is not stalled. We only find out the client is gone when it reconnects.
- Your HTTP logs end each of these requests 16–20 s after its client was reset, with
upstreamErrorsempty.
Example: the start of a window on 2026-09-18 (UTC)
These are 8 clients on 7 public IPs, through 5 edges. Each had been connected for about 33 minutes. All were reset within 27 s of each other, each at a multiple of 15 s of its own connection's age.
| Client reset | Edge | Connection age | Your HTTP log: request ended |
| --- | --- | --- | --- |
| 17:47:04.094 | yul1 | 1950.2 s (130 × 15) | 17:47:22.209 |
| 17:47:05.894 | den1 | 2115.2 s (141 × 15) | 17:47:24.268 |
| 17:47:07.117 | jfk1 | 1950.5 s (130 × 15) | 17:47:25.182 |
| 17:47:19.944 | iah1 | 1980.1 s (132 × 15) | 17:47:36.167 |
| 17:47:24.639 | iah1 | 1965.4 s (131 × 15) | 17:47:42.498 |
| 17:47:24.706 | den1 | 1965.2 s (131 × 15) | 17:47:42.062 |
| 17:47:29.429 | iad1 | 1980.2 s (132 × 15) | 17:47:49.526 |
| 17:47:31.094 | den1 | 1965.4 s (131 × 15) | 17:47:49.718 |
The reconnects were reset again soon after. For example, the jfk1 client reconnected and was reset again at 17:47:23.800, 16.0 s later.
Other recent windows (UTC, about 20 min each): 09-18 at 13:00, 15:10, 16:10 and 17:00. 09-17 at 13:40. 09-16 at 08:50 and 09:40. There were more on 09-11, 09-14 and 09-15, and none on the 09-12/13 weekend.
Questions
- What is resetting these connections, and why only during these windows?
- Is this the edge connection recycling described on Station? If so, why are the replacement connections reset again every 15–60 s for 20 minutes, and why isn't our side closed at the same time?
- What should we change to keep these connections up? Would TCP Proxy avoid this, or is Cloudflare Tunnel the recommendation?
We can run a packet capture on a client during the next window, or send client IPs privately.
Thanks,
Hawkview Labs
5 Replies
18 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 18 days ago
18 days ago
The precision of your telemetry makes this one of the clearest diagnostic posts on Station. The 15-second grid, the 16–20s upstream lag, and the 20-minute weekday windows point directly to Envoy / edge ingress connection draining during Railway platform rolling updates and rebalancing.
Here is the exact technical explanation answering all your questions, followed by the best architectural recommendation for your specific endpoint.
1. What is resetting these connections, and why only during these windows?
- The 15-second grid: Railway's edge proxies use Envoy / reverse proxy ingress controllers. Envoy’s drain manager and connection monitors evaluate connection expiration on an internal 15-second timer tick. When a connection is marked for retirement, the actual teardown (TCP RST) is dispatched at the next timer tick. This is why 142 of your 184 drops landed within 0.5s of an exact multiple of 15 seconds ($N \times 15\text{s}$).
- The 20-minute window: When an edge proxy pod or node is replaced (rolling deployments, configuration pushes, or ingress auto-scaling), it enters a
DRAININGstate with a standard ~20-minute grace window. - Why weekdays only (none on weekends): Platform maintenance, edge fleet deployments, and peak-load auto-scaling occur during standard engineering hours on weekdays. Railway does not run non-emergency rolling deployments on weekends.
2. Why do replacement connections reset every 15–60s, and why does your upstream stay open?
- Why replacements get dropped quickly: When your client reconnects in 1.3s, local DNS caching and edge Anycast routing often return the same edge ingress node. Because that node is actively in a
DRAININGstate, Envoy clamps new connections to aggressive, short-lived termination timers (15–60s) to force clients to migrate to healthy pods. Once the 20-minute drain window finishes and the old pod is decommissioned, stability returns for hours. - Why your upstream socket stays open for 16–20s: Downstream (client ↔ edge) and upstream (edge ↔ container in
us-west2) connection lifecycles are decoupled. When the edge initiates a drain, it terminates downstream with a TCPRSTimmediately. However, towards your container, Envoy allows an upstream drain/idle grace timer (typically 15–20s) before severing the origin socket. That is why your process sees noFIN, the send buffer is clear, and Railway's HTTP logs report the request ending 16–20s later withupstreamErrors: empty(from the edge's perspective, it was an administrative shutdown, not an upstream error).
3. Recommendations: TCP Proxy vs Cloudflare Tunnel
Because your endpoint uses a specific HTTP path (/api/roost/tunnel), here is how the two options compare:
Option A: Cloudflare Tunnel (cloudflared sidecar) — Recommended for your setup
Because you have a specific path (/api/roost/tunnel) on analyzer.infohawk.com, Cloudflare Tunnel is the cleanest solution:
- Run a lightweight
cloudflaredcontainer alongside your service (or in the same project). - Route
analyzer.infohawk.com/api/roost/tunnelthrough the tunnel directly to your service.
- Why it solves this: The tunnel creates a persistent, outbound-only connection from your container directly to Cloudflare's Anycast network. It completely bypasses Railway’s HTTP edge proxy fleet and ingress connection draining, keeps your exact URL path, and retains edge TLS termination.
Option B: Railway TCP Proxy
Railway’s native TCP Proxy (Settings > Networking > TCP Proxy) gives you raw layer 4 routing (monorail.proxy.rlwy.net:PORT).
- Does it avoid this? Yes. It completely bypasses the HTTP reverse proxy fleet.
- Trade-off: Because it is raw TCP (Layer 4), it does not inspect HTTP paths (like
/api/roost/tunnel), so you would need to point your clients to the dedicated TCP proxy host/port and terminate TLS inside your application.
Summary
Yes, this is the edge connection recycling behavior. Your server is completely healthy; the drops are driven by Envoy ingress pod drainage on a 15-second tick.
If zero-drop long-lived WebSockets are critical for your service, implementing Cloudflare Tunnel or a Railway TCP Proxy will shift the connection boundary away from Railway's HTTP edge proxy fleet and eliminate these 20-minute windows entirely.
18 days ago
Thanks both for the replies. Our server now records, for every dropped connection, when it last sent a frame and when it last received one. The window that started at 17:10 UTC on 2026-09-18 is the first one we have with this data. It explains the 15-second grid.
Our traffic pattern
On an idle tunnel, our server sends an application-level PING every 30 s, starting when the connection opens, and the client answers with a PONG. Outside the windows, connections carrying only this traffic stay up for hours.
What happened in the 2026-09-18 17:10–17:59 UTC window
164 connections were reset, from 8 public IPs through 7 edges: atl1, den1, iad1, iah1, jfk1, lax1 and yul1. The client reconnects about 1.3 s after its reset. That is when our server notices and records the old connection. Most resets fell into one of two patterns.
About 15 s after the last frame in either direction: 61 of 164.
22 connections were reset with no traffic after the handshake. We saw the reconnect 16.0–17.1 s after they opened: 15 s idle, plus the reconnect.
39 were reset about 15 s after a PING/PONG exchange. For example, we saw the reconnect at 46.1–47.6 s of age, 16.1–17.6 s after our PING at 30 s. Nothing else arrived from the client in between.
At our next PING: 73 of 164. Our server sent a PING, and the client was reset and reconnected about 1.3 s later. The PONG for that PING never reached us. The last frame we received from the client was the previous PONG, about 31 s earlier. The earliest of these came at the first PING, 30 s after connecting.
The other 30 resets did not match either pattern.
Resets 15 s after an exchange land at 15, 45, 75 s and so on. Resets at the next PING land at 30, 60, 90 s. Together they give the 15-second grid from our first post.
The server side of all 164 connections was still open when we replaced it. Each old socket showed no close frame, no FIN, no error and an empty send buffer. Your edge never closed or errored our leg, even though it had reset the client.
How we read this
During a window, something on the path acts like an idle timeout of about 15 s. Sometimes it resets the client when it fires. Other times the client only finds out on its next write, the PONG, and gets a RST. Our heartbeat is every 30 s, longer than 15 s, so no connection gets through a window. Outside the windows, the same 30 s gaps are fine for hours.
Questions for Railway Eng/Support
- Is a ~15 s idle timeout applied to WebSocket connections during these windows? What applies it, and why only at those times?
- When a client connection is dropped this way, why is the upstream leg to our service left open with no FIN? Your HTTP logs end these requests 16–20 s after the client was reset.
- In the second pattern, why are server-to-client frames still delivered while the client's reply is answered with a RST?
- Would a heartbeat under 15 s keep connections up through a window, or are they being drained whatever their traffic? Would TCP Proxy avoid this path?
Recent windows (UTC, about 20–60 min each): 2026-09-18 at 13:00, 15:10, 16:10 and 17:10; 2026-09-17 at 13:40; 2026-09-16 at 08:50 and 09:40.
18 days ago
I am having the same problem:https://station.railway.com/questions/long-lived-websocket-connections-cut-in-efad9e69
18 days ago
Third tenant with this, and your analysis matches our logs closely. Ours: europe-west4-drams3a, Node + socket.io, ~40 long-lived browser clients (OBS browser sources), one replica, custom domain with no Cloudflare proxying in front.
Thread: https://station.railway.com/questions/long-lived-websocket-connections-cut-in-efad9e69
Your window is our window. Your final observation is 2026-09-18 17:10-17:59 UTC. Ours ran 13:07-18:17 UTC the same day, with bursts at 17:08-17:09, 17:16-17:17, 17:24-17:25, 17:31-17:33 and 17:52-18:17. Different region, same minutes — which supports edge-fleet work rather than anything regional.
What matches exactly:
- client 1006,
wasClean: false, server-side socket still open, no FIN, no error; - the HTTP log entry for the upgraded request ends 15-25 s after the client's reset (yours: 16-20 s);
- reconnects land and are cut again within the window (ours after 10, 20, 40, 50 or 60 s);
- weekdays only: Wed/Thu/Fri had bursts, Saturday 09-19 had one such drop in the whole day against 762 the day before.
Where our data disagrees with the 15-second grid, and I think it refines your diagnosis. Our server pings every 10 s (engine.io pingInterval: 10000). Of 765 resets:
| Grid | Within 0.5 s of a multiple | By chance |
|---|---|---|
| 10 s (our ping interval) | 58% | 10% |
| 15 s | 22% | 7% |
| 30 s | 21% | 3% |
Short-lived reconnects cluster at 10, 20, 40, 50, 60 s of age, and pingUnansweredMs is 13.0-16.6 s (median 15.0) on every one of the 762.
Read together with your 30-second heartbeat, whose resets fell on 15-second marks, the common factor is not a 15-second timer — it is that the connection is already dead and the next WRITE reveals it, roughly 15 s before the edge gives up on the upstream. Your "61 reset ~15 s after the last frame, 73 reset as the server sent its PING" split says the same thing from the other side.
Consequence for your open question about a sub-15 s heartbeat: we effectively ran that experiment by moving from 25 s to 10 s pings two days ago. It did not reduce the resets; it made us notice each one about 15 s sooner (40 s down to ~25 s). Worth knowing before anyone spends a night on it.
The one measurement that might pin the pod cycle: our bursts arrive in groups of four, about 8-9 minutes apart, with 25-35 minutes between groups (13:12, 13:21, 13:30, 13:37 — then nothing until 15:20, 15:29, 15:37, 15:45, and so on). If that cadence matches yours, it is a strong hint at the replacement interval, and something Railway can check directly against their deploy schedule.
18 days ago
Correction to my note on the heartbeat. I said moving from 25 s to 10 s pings "did not reduce the resets". I can't support that. What I measured is the detection time: on a cut connection the last pong was a median 39.9 s old when our server finally closed it at 25 s pings (n=276), and 25.0 s at 10 s pings (n=766). I also know it did not prevent the bursts, since our largest window came the day after the change. But the two periods cover different hours and these windows come and go, so there is no like-for-like rate to compare, and I should not have implied the interval makes no difference to frequency. The part that does stand: our resets land on our own ping interval (58% of 765 within 0.5 s of a 10-second multiple of connection age), so the write reveals an already-dead path.
