Skip to content
iOS clients lose the connection right after upload; backend completes every request (edge lax1, since Oct 20)

22 days ago

Service: ctvd-recognizer (49d2fc23-80f1-4069-8ad3-9ed355c82ae9) in project protective-delight (b1c7d2c0-6efd-4128-acd4-4f0975cc6e16), environment production, region us-west2, Hobby plan, public domain api.cubesnap.app. Response headers show server: railway-hikari and x-railway-edge: lax1.

Symptom: since about 2026-10-20, a growing share of requests from our iOS app (Apple URLSession, HTTP/2) to POST /api/recognize (multipart body of 400 to 800 KB, JSON response after about 2 s) fail on the phone with NSURLErrorNetworkConnectionLost (-1005). Our backend receives the full body and processes every one of these requests: we can match each client failure to a backend event 0 to 3 s later, with normal latency (p50 1.9 s, p90 2.7 s; the lost requests are not the slow ones). So the client is cut off right after the upload completes, before the response is written. In two cases the backend got BrokenPipeError while writing the response, at 2026-10-25T00:48:19Z (peer 100.64.0.23:27714 as seen by the container). Android (OkHttp) and browser clients show zero such failures in the same period (19 Android sessions, 56 web devices since 10-25), and the iOS networking code has not changed.

Affected iOS requests per day: 10-15 to 10-19: 0, 1, 1, 2, 0. 10-20: 4. 10-21: 7. 10-22: 8. 10-23: 7. 10-24: 5. 10-25: 12 (8 devices out of 43 sessions). 10-26: 14 by 16:00 UTC (5 devices out of 36 sessions).

Example timestamps (UTC): one device failed 8 consecutive times between 2026-10-26 01:17:58 and 01:20:30, all 8 processed by the backend. 2026-10-25 00:48:14 and 00:48:20 are the pair that produced the broken-pipe tracebacks.

Questions: did the edge in front of this service change around Oct 20 (hikari rollout or incident remediation) in a way that affects HTTP/2 request bodies from Apple's URLSession? Can you check edge logs for this service at the timestamps above for stream resets, GOAWAY, or connection closes toward the client? Happy to provide more timestamps or run a test build.

Solved$10 Bounty

3 Replies

Railway
BOT

22 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 22 days ago


sundayabasi684-byte
Top 5% Contributor

21 days ago

Railway edge HTTP/2 RST_STREAM/GOAWAY after multipart upload: backend processes request, client disconnects before response, BrokenPipeError while writing response, Apple URLSession NSURLErrorNetworkConnectionLost (-1005), x-railway-edge lax1, railway-hikari

The request body is fully uploaded and the backend processes it normally, but the client loses the connection immediately after upload completes. The server logs BrokenPipeError while writing the response, which indicates the peer closed the connection before the response was delivered. This matches an edge/proxy HTTP/2 stream reset or connection close rather than a backend business-logic bug.

Affected requests are iOS Apple URLSession over HTTP/2; Android/browser are not seeing the same issue. The issue started around 2026-10-20 and increased afterward. Response headers show server: railway-hikari and x-railway-edge: lax1.

This is consistent with:

HTTP/2 stream reset

RST_STREAM

GOAWAY

edge-side connection close

upstream response write failure after client disconnect

Relevant timestamps:

2026-10-25T00:48:14Z

2026-10-25T00:48:20Z

2026-10-26 01:17:58 to 01:20:30 UTC

Expected: the client receives the response after upload completes. Actual: the connection is lost before the response is delivered.

This is likely an edge / proxy HTTP/2 issue on the Railway path, not an application bug in /api/recognize.


sundayabasi684-byte
Top 5% Contributor

21 days ago

Have you tried this


20 days ago

Update from the reporter (Oct 28): this was on our side, not Railway's. Our 1.0.18 iOS release added per-request diagnostics, and every failure it has recorded since carries NSURLErrorCancelled (-999), not -1005. Our wait screen shows a Cancel button after 4 seconds, and the app was logging a user's Cancel tap as a transport failure, so the "connection lost" class was mostly people cancelling slow uploads: 73 of the 82 iOS failures since Oct 20 sit at 3 to 8 seconds, and the backend finishing 0 to 3 seconds later was simply the upload completing after they bailed. The Android app never logged its cancels, which produced the apparent iOS-only pattern. Two genuine -1005 cases at about 35 seconds remain in the sample, rare enough that we are not asking Railway to look further. We are fixing the telemetry and the wait screen in our next release. Sorry for the noise, and thanks; resolving this thread.


Status changed to Solved jeffhuber • 20 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...