Intermittent HTTP POST body stream stalls behind Railway ingress
calebrichter
PROOP

a month ago

We are experiencing intermittent severe timeouts on public HTTP POST requests to our backend service. Clients receive zero bytes and time out after 20s+. GET health checks remain consistently fast.

We isolated this with request-bound telemetry at the very start of our Express app, before body parsing.

Evidence

  • Headers arrive immediately, but body stream stalls.

    Our app logs graphql.request.received as soon as the request enters Express. On slow requests, data and end on the request stream do not fire for many seconds. At 5s we log bodyBytes=0.

  • Node event loop is healthy.

    We monitor with perf_hooks.monitorEventLoopDelay(). During stalls, event-loop delay remains normal, typically around 20-30ms with max usually below ~75ms. The process is not blocked.

  • GraphQL/database/resolvers are not involved.

    The issue reproduces with:

    query Noop { __typename }

    Request bodies are tiny, around 125-148 bytes. The stall happens before GraphQL execution and before Prisma/database access.

  • Bypassing public ingress is instant.

    From inside the Railway container, local POSTs to:

    http://127.0.0.1:443/api/graphql

    return consistently in ~0.6-2ms. Public POSTs to the same GraphQL endpoint intermittently hang.

  • Scaling to one replica did not fix it.

    We temporarily set production to a single replica, so this does not appear to be inter-replica load balancing.

  • We also tried backend mitigations:

    • bypassed our global express.json() for /api/graphql

    • set Node HTTP server:

      httpServer.keepAliveTimeout = 65000;

      httpServer.headersTimeout = 66000;

    The issue persisted.

Example log pattern

graphql.request.received

durationMs=0 contentLength=127

graphql.request.still_in_flight

durationMs=5000 bodyBytes=0 eventLoopDelayMaxMs=22

graphql.request.body_first_chunk

durationMs=93625

graphql.request.body_end

durationMs=93625 bodyBytes=127

graphql.response.finished

durationMs=93627 statusCode=200

Another example:

graphql.request.still_in_flight

durationMs=5001 bodyBytes=0 contentLength=146 eventLoopDelayMaxMs=23

graphql.request.body_first_chunk

durationMs=100410

graphql.request.body_end

durationMs=100410 bodyBytes=146

graphql.response.finished

durationMs=100411 statusCode=200

Fresh public probe results

probe 1: timeout after 20s, 0 bytes received

probe 2: 200 in 8.5s

probe 3: 200 in 0.18s

probe 4: 200 in 14.3s

probe 5: timeout after 20s, 0 bytes received

probe 6: timeout after 20s, 0 bytes received

probe 7: timeout after 20s, 0 bytes received

probe 8: 200 in 2.1s

Service details

  • Project ID: d46c9f58-799a-45e8-9686-54cc7da4f973
  • Environment ID: c491af8d-1625-40e6-badd-fcbae1a59f88
  • Service ID: 4e6029c5-82ee-4f5a-8936-1e55a45e0f15
  • Service: ptw-backend
  • Public endpoint: https://admin.practicingtheway.org/api/graphql
  • Runtime: Railway V2
  • Current replicas: 1

Can Railway investigate the ingress/proxy path for this service? From the app’s perspective, request headers are delivered immediately, but the POST body is delayed or withheld for 20-100s despite tiny

Content-Length values. We’d like to know whether there is an ingress proxy/body-streaming, buffering, or keep-alive connection reuse issue on this routing path.

$20 Bounty

0 Replies

Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...