a month ago
We are experiencing intermittent severe timeouts on public HTTP POST requests to our backend service. Clients receive zero bytes and time out after 20s+. GET health checks remain consistently fast.
We isolated this with request-bound telemetry at the very start of our Express app, before body parsing.
Evidence
-
Headers arrive immediately, but body stream stalls.
Our app logs graphql.request.received as soon as the request enters Express. On slow requests, data and end on the request stream do not fire for many seconds. At 5s we log bodyBytes=0.
-
Node event loop is healthy.
We monitor with perf_hooks.monitorEventLoopDelay(). During stalls, event-loop delay remains normal, typically around 20-30ms with max usually below ~75ms. The process is not blocked.
-
GraphQL/database/resolvers are not involved.
The issue reproduces with:
query Noop { __typename }
Request bodies are tiny, around 125-148 bytes. The stall happens before GraphQL execution and before Prisma/database access.
-
Bypassing public ingress is instant.
From inside the Railway container, local POSTs to:
http://127.0.0.1:443/api/graphql
return consistently in ~0.6-2ms. Public POSTs to the same GraphQL endpoint intermittently hang.
-
Scaling to one replica did not fix it.
We temporarily set production to a single replica, so this does not appear to be inter-replica load balancing.
-
We also tried backend mitigations:
-
bypassed our global express.json() for /api/graphql
-
set Node HTTP server:
httpServer.keepAliveTimeout = 65000;
httpServer.headersTimeout = 66000;
The issue persisted.
-
Example log pattern
graphql.request.received
durationMs=0 contentLength=127
graphql.request.still_in_flight
durationMs=5000 bodyBytes=0 eventLoopDelayMaxMs=22
graphql.request.body_first_chunk
durationMs=93625
graphql.request.body_end
durationMs=93625 bodyBytes=127
graphql.response.finished
durationMs=93627 statusCode=200
Another example:
graphql.request.still_in_flight
durationMs=5001 bodyBytes=0 contentLength=146 eventLoopDelayMaxMs=23
graphql.request.body_first_chunk
durationMs=100410
graphql.request.body_end
durationMs=100410 bodyBytes=146
graphql.response.finished
durationMs=100411 statusCode=200
Fresh public probe results
probe 1: timeout after 20s, 0 bytes received
probe 2: 200 in 8.5s
probe 3: 200 in 0.18s
probe 4: 200 in 14.3s
probe 5: timeout after 20s, 0 bytes received
probe 6: timeout after 20s, 0 bytes received
probe 7: timeout after 20s, 0 bytes received
probe 8: 200 in 2.1s
Service details
- Project ID: d46c9f58-799a-45e8-9686-54cc7da4f973
- Environment ID: c491af8d-1625-40e6-badd-fcbae1a59f88
- Service ID: 4e6029c5-82ee-4f5a-8936-1e55a45e0f15
- Service: ptw-backend
- Public endpoint: https://admin.practicingtheway.org/api/graphql
- Runtime: Railway V2
- Current replicas: 1
Can Railway investigate the ingress/proxy path for this service? From the app’s perspective, request headers are delivered immediately, but the POST body is delayed or withheld for 20-100s despite tiny
Content-Length values. We’d like to know whether there is an ingress proxy/body-streaming, buffering, or keep-alive connection reuse issue on this routing path.