a month ago
Service: draftdog-web, environment production, deployment 089b5962-ecdd-4acd-8708-cc28082007c9 (5 replicas). Edge: x-railway-edge: jfk1, server: railway-hikari. Endpoint: POST /api/chat/v2/conversations/{id}/message — SSE (text/event-stream), 10s keepalive comments, streams normally 5-90s.
What we observed (2026-07-18, ~08:00–23:00 UTC): the edge actively terminated in-flight SSE responses. HTTP/1.1 clients received a clean FIN mid-chunked-body (curl exit 18, "transfer closed with outstanding read data remaining"); HTTP/2 clients received RST_STREAM (CANCEL, err 8). Kills hit multiple concurrent streams from one client at the same instant — client-side failure timestamps identical to the millisecond across independent TCP connections — in sweeps at 21:30:26.450Z, 21:31:00.58Z, 21:32:45Z, 21:33:00Z, 21:33:56Z, 21:34:08Z, 21:34:15Z, 21:34:30Z, 21:34:56Z, and 21:35:08Z. Stream ages at kill ranged 0.2s–75s — far below the documented 15-min / 5-min-no-data thresholds, and many kills happened with data actively flowing.
The origin was healthy throughout: every request logged HTTP 200, uvicorn only observed the closed transport 11–28s after the client-side death, responses completed server-side and persisted to our database. No deploys, restarts, or resource pressure in the window. Control test: 4 concurrent 90s SSE streams from the same client/network to a non-Railway SSE host ran flawlessly (~4MB each, zero truncation), and TLS chain verified genuine — the local path is exonerated.
Concrete artifact for correlation: killed stream request ID dGatkZCcSAmP-bDgwoOzXw, window 22:32:14–22:32:56Z, HTTP/2, RST_STREAM(CANCEL) at 41.4s into the response. We can supply origin-side request UUIDs + exact timestamps for every request in each sweep window on request (e.g. 8 requests logged 21:30:21–21:30:30Z around the first sweep).
Likely related: this looks like the same underlying edge bug as the open thread https://station.railway.com/questions/edge-hikari-intermittently-sends-rst-s-3ed8fbb0 (hikari RST_STREAM(CANCEL) after origin 200, since 07-16) — with the additional datum that HTTP/1.1 is affected too (clean FIN instead of RST), so the bug is not confined to the H2 path.
Questions: (1) Were jfk1 edge instances recycling/draining on 07-18? (2) Does hikari apply any per-client concurrent-stream reaping or undocumented in-flight termination? (3) Can you trace jfk1 edge logs at the sweep timestamps + the request ID above and tell us whether the edge initiated the terminations?
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Seeing the same bug on the same edge (jfk1, railway-hikari) on service evidence-os-production. Independent reproduction below.
In our case the truncations are strongly size-correlated: responses under ~2KB never fail, responses over ~8KB fail ~1 in 4, and every truncation clusters tightly at 8398–8762 bytes. Client gets a clean EOF (curl exit 0), origin logs HTTP 200 and persists the full response every time. Bypassing our Cloudflare Worker (direct to *.up.railway.app) still truncates, so it's the edge, not our CDN.
This may be the same root cause as your sweep observation — an edge instance dropping the connection mid-body. We have a standalone FastAPI reproduction (any endpoint streaming >8KB of text/event-stream over 10s+) and can supply x-railway-request-id values for confirmed truncations.
Full writeup with A/B data ruling out CDN, byte-clustering, and size-correlation available on request.