3 months ago
We are experiencing intermittent severe timeouts on public HTTP POST requests to our backend service. Clients receive zero bytes and time out after 20s+. GET health checks remain consistently fast.
We isolated this with request-bound telemetry at the very start of our Express app, before body parsing.
Evidence
-
Headers arrive immediately, but body stream stalls.
Our app logs graphql.request.received as soon as the request enters Express. On slow requests, data and end on the request stream do not fire for many seconds. At 5s we log bodyBytes=0.
-
Node event loop is healthy.
We monitor with perf_hooks.monitorEventLoopDelay(). During stalls, event-loop delay remains normal, typically around 20-30ms with max usually below ~75ms. The process is not blocked.
-
GraphQL/database/resolvers are not involved.
The issue reproduces with:
query Noop { __typename }
Request bodies are tiny, around 125-148 bytes. The stall happens before GraphQL execution and before Prisma/database access.
-
Bypassing public ingress is instant.
From inside the Railway container, local POSTs to:
http://127.0.0.1:443/api/graphql
return consistently in ~0.6-2ms. Public POSTs to the same GraphQL endpoint intermittently hang.
-
Scaling to one replica did not fix it.
We temporarily set production to a single replica, so this does not appear to be inter-replica load balancing.
-
We also tried backend mitigations:
-
bypassed our global express.json() for /api/graphql
-
set Node HTTP server:
httpServer.keepAliveTimeout = 65000;
httpServer.headersTimeout = 66000;
The issue persisted.
-
Example log pattern
graphql.request.received
durationMs=0 contentLength=127
graphql.request.still_in_flight
durationMs=5000 bodyBytes=0 eventLoopDelayMaxMs=22
graphql.request.body_first_chunk
durationMs=93625
graphql.request.body_end
durationMs=93625 bodyBytes=127
graphql.response.finished
durationMs=93627 statusCode=200
Another example:
graphql.request.still_in_flight
durationMs=5001 bodyBytes=0 contentLength=146 eventLoopDelayMaxMs=23
graphql.request.body_first_chunk
durationMs=100410
graphql.request.body_end
durationMs=100410 bodyBytes=146
graphql.response.finished
durationMs=100411 statusCode=200
Fresh public probe results
probe 1: timeout after 20s, 0 bytes received
probe 2: 200 in 8.5s
probe 3: 200 in 0.18s
probe 4: 200 in 14.3s
probe 5: timeout after 20s, 0 bytes received
probe 6: timeout after 20s, 0 bytes received
probe 7: timeout after 20s, 0 bytes received
probe 8: 200 in 2.1s
Service details
- Project ID: d46c9f58-799a-45e8-9686-54cc7da4f973
- Environment ID: c491af8d-1625-40e6-badd-fcbae1a59f88
- Service ID: 4e6029c5-82ee-4f5a-8936-1e55a45e0f15
- Service: ptw-backend
- Public endpoint: https://admin.practicingtheway.org/api/graphql
- Runtime: Railway V2
- Current replicas: 1
Can Railway investigate the ingress/proxy path for this service? From the app’s perspective, request headers are delivered immediately, but the POST body is delayed or withheld for 20-100s despite tiny
Content-Length values. We’d like to know whether there is an ingress proxy/body-streaming, buffering, or keep-alive connection reuse issue on this routing path.
1 Replies
3 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 3 months ago
13 days ago
Your telemetry already rules out the GraphQL resolver, Prisma, body-parser CPU work, event-loop starvation, and replica balancing. The decisive observation is that Express receives the headers immediately but the first request-body byte arrives 9–100 seconds later. That puts the delay between the client upload and the app-facing ingress stream.
I would run one bounded matrix and attach the results with UTC timestamps plus one unique request ID per attempt:
payload='{"query":"query Noop { __typename }"}'
curl --http1.1 --no-keepalive -H 'Connection: close' -H 'Expect:' \
-H 'Content-Type: application/json' -H "X-Debug-Id: fresh-$(date +%s)" \
--data-binary "$payload" \
-w '\nconnect=%{time_connect} tls=%{time_appconnect} pretransfer=%{time_pretransfer} start=%{time_starttransfer} total=%{time_total} upload=%{size_upload}\n' \
https://admin.practicingtheway.org/api/graphqlRepeat the same request through (a) the custom domain and (b) the Railway-provided domain. Then compare a fresh HTTP/1.1 connection with the normal pooled client, and compare a fixed Content-Length body with chunked transfer. Keeping Expect: empty removes 100-continue from the test. If the fresh Connection: close requests are consistently fast while pooled requests stall, that isolates connection reuse. If only one hostname stalls, resolve both hostnames and retain the edge IP/region from each run; that isolates the custom-domain/edge route. If fixed length is fast but chunked transfer stalls, the discriminator is body framing at ingress.
At the app boundary, log only these fields for the matching X-Debug-Id: header-received timestamp, remote address, HTTP version, content-length, transfer-encoding, first-byte timestamp, end timestamp, and aborted/close events. Avoid logging the body or auth headers. The client trace and app log then form a single timeline Railway can correlate against edge logs.
I would not tune keepAliveTimeout or headersTimeout further: both act after the request reaches Node and do not explain a delayed first body byte with a healthy event loop. As a temporary mitigation, use the consistently-fast hostname/path from the matrix, disable connection pooling for affected POSTs, and apply a client retry only when zero response bytes were received and the operation is idempotent (or protected by an idempotency key).
The minimal escalation package is therefore: 5–10 UTC timestamps, X-Debug-Id, hostname, resolved edge IP, HTTP version/framing, curl timing fields, and the paired app first-byte delay. That should let Railway identify whether one ingress edge or reused upstream connection is holding the upload stream.