22 days ago
Service: ctvd-recognizer (49d2fc23-80f1-4069-8ad3-9ed355c82ae9) in project protective-delight (b1c7d2c0-6efd-4128-acd4-4f0975cc6e16), environment production, region us-west2, Hobby plan, public domain api.cubesnap.app. Response headers show server: railway-hikari and x-railway-edge: lax1.
Symptom: since about 2026-10-20, a growing share of requests from our iOS app (Apple URLSession, HTTP/2) to POST /api/recognize (multipart body of 400 to 800 KB, JSON response after about 2 s) fail on the phone with NSURLErrorNetworkConnectionLost (-1005). Our backend receives the full body and processes every one of these requests: we can match each client failure to a backend event 0 to 3 s later, with normal latency (p50 1.9 s, p90 2.7 s; the lost requests are not the slow ones). So the client is cut off right after the upload completes, before the response is written. In two cases the backend got BrokenPipeError while writing the response, at 2026-10-25T00:48:19Z (peer 100.64.0.23:27714 as seen by the container). Android (OkHttp) and browser clients show zero such failures in the same period (19 Android sessions, 56 web devices since 10-25), and the iOS networking code has not changed.
Affected iOS requests per day: 10-15 to 10-19: 0, 1, 1, 2, 0. 10-20: 4. 10-21: 7. 10-22: 8. 10-23: 7. 10-24: 5. 10-25: 12 (8 devices out of 43 sessions). 10-26: 14 by 16:00 UTC (5 devices out of 36 sessions).
Example timestamps (UTC): one device failed 8 consecutive times between 2026-10-26 01:17:58 and 01:20:30, all 8 processed by the backend. 2026-10-25 00:48:14 and 00:48:20 are the pair that produced the broken-pipe tracebacks.
Questions: did the edge in front of this service change around Oct 20 (hikari rollout or incident remediation) in a way that affects HTTP/2 request bodies from Apple's URLSession? Can you check edge logs for this service at the timestamps above for stream resets, GOAWAY, or connection closes toward the client? Happy to provide more timestamps or run a test build.
3 Replies
22 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 22 days ago
21 days ago
Railway edge HTTP/2 RST_STREAM/GOAWAY after multipart upload: backend processes request, client disconnects before response, BrokenPipeError while writing response, Apple URLSession NSURLErrorNetworkConnectionLost (-1005), x-railway-edge lax1, railway-hikari
The request body is fully uploaded and the backend processes it normally, but the client loses the connection immediately after upload completes. The server logs BrokenPipeError while writing the response, which indicates the peer closed the connection before the response was delivered. This matches an edge/proxy HTTP/2 stream reset or connection close rather than a backend business-logic bug.
Affected requests are iOS Apple URLSession over HTTP/2; Android/browser are not seeing the same issue. The issue started around 2026-10-20 and increased afterward. Response headers show server: railway-hikari and x-railway-edge: lax1.
This is consistent with:
HTTP/2 stream reset
RST_STREAM
GOAWAY
edge-side connection close
upstream response write failure after client disconnect
Relevant timestamps:
2026-10-25T00:48:14Z
2026-10-25T00:48:20Z
2026-10-26 01:17:58 to 01:20:30 UTC
Expected: the client receives the response after upload completes. Actual: the connection is lost before the response is delivered.
This is likely an edge / proxy HTTP/2 issue on the Railway path, not an application bug in /api/recognize.
21 days ago
Have you tried this
20 days ago
Update from the reporter (Oct 28): this was on our side, not Railway's. Our 1.0.18 iOS release added per-request diagnostics, and every failure it has recorded since carries NSURLErrorCancelled (-999), not -1005. Our wait screen shows a Cancel button after 4 seconds, and the app was logging a user's Cancel tap as a transport failure, so the "connection lost" class was mostly people cancelling slow uploads: 73 of the 82 iOS failures since Oct 20 sit at 3 to 8 seconds, and the backend finishing 0 to 3 seconds later was simply the upload completing after they bailed. The Android app never logged its cancels, which produced the apparent iOS-only pattern. Two genuine -1005 cases at about 35 seconds remain in the sample, rare enough that we are not asking Railway to look further. We are fixing the telemetry and the wait screen in our next release. Sorry for the noise, and thanks; resolving this thread.
Status changed to Solved jeffhuber • 20 days ago