15 hours ago
Service: ctvd-recognizer (49d2fc23-80f1-4069-8ad3-9ed355c82ae9) in project protective-delight (b1c7d2c0-6efd-4128-acd4-4f0975cc6e16), environment production, region us-west2, Hobby plan, public domain api.cubesnap.app. Response headers show server: railway-hikari and x-railway-edge: lax1.
Symptom: since about 2026-10-20, a growing share of requests from our iOS app (Apple URLSession, HTTP/2) to POST /api/recognize (multipart body of 400 to 800 KB, JSON response after about 2 s) fail on the phone with NSURLErrorNetworkConnectionLost (-1005). Our backend receives the full body and processes every one of these requests: we can match each client failure to a backend event 0 to 3 s later, with normal latency (p50 1.9 s, p90 2.7 s; the lost requests are not the slow ones). So the client is cut off right after the upload completes, before the response is written. In two cases the backend got BrokenPipeError while writing the response, at 2026-10-25T00:48:19Z (peer 100.64.0.23:27714 as seen by the container). Android (OkHttp) and browser clients show zero such failures in the same period (19 Android sessions, 56 web devices since 10-25), and the iOS networking code has not changed.
Affected iOS requests per day: 10-15 to 10-19: 0, 1, 1, 2, 0. 10-20: 4. 10-21: 7. 10-22: 8. 10-23: 7. 10-24: 5. 10-25: 12 (8 devices out of 43 sessions). 10-26: 14 by 16:00 UTC (5 devices out of 36 sessions).
Example timestamps (UTC): one device failed 8 consecutive times between 2026-10-26 01:17:58 and 01:20:30, all 8 processed by the backend. 2026-10-25 00:48:14 and 00:48:20 are the pair that produced the broken-pipe tracebacks.
Questions: did the edge in front of this service change around Oct 20 (hikari rollout or incident remediation) in a way that affects HTTP/2 request bodies from Apple's URLSession? Can you check edge logs for this service at the timestamps above for stream resets, GOAWAY, or connection closes toward the client? Happy to provide more timestamps or run a test build.
2 Replies
15 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 15 hours ago
14 hours ago
Railway edge HTTP/2 RST_STREAM/GOAWAY after multipart upload: backend processes request, client disconnects before response, BrokenPipeError while writing response, Apple URLSession NSURLErrorNetworkConnectionLost (-1005), x-railway-edge lax1, railway-hikari
The request body is fully uploaded and the backend processes it normally, but the client loses the connection immediately after upload completes. The server logs BrokenPipeError while writing the response, which indicates the peer closed the connection before the response was delivered. This matches an edge/proxy HTTP/2 stream reset or connection close rather than a backend business-logic bug.
Affected requests are iOS Apple URLSession over HTTP/2; Android/browser are not seeing the same issue. The issue started around 2026-10-20 and increased afterward. Response headers show server: railway-hikari and x-railway-edge: lax1.
This is consistent with:
HTTP/2 stream reset
RST_STREAM
GOAWAY
edge-side connection close
upstream response write failure after client disconnect
Relevant timestamps:
2026-10-25T00:48:14Z
2026-10-25T00:48:20Z
2026-10-26 01:17:58 to 01:20:30 UTC
Expected: the client receives the response after upload completes. Actual: the connection is lost before the response is delivered.
This is likely an edge / proxy HTTP/2 issue on the Railway path, not an application bug in /api/recognize.
2 hours ago
Have you tried this