Intermittent 30s-200s API response delays with low CPU/memory
miggalvez
HOBBYOP

3 months ago

We’re seeing what looks like the same pattern on our Railway-hosted Express/Node backend.

Region/platform: Railway app service with MongoDB Atlas backend

Started noticing: after our June 3 deploy, but symptoms continued well beyond normal warmup.

Symptoms:

  • Intermittent very slow API responses, especially authenticated writes/actions.
  • POST /api/auth/login took 15s and later 42s.
  • POST /api/forms/:id/publish took 42s.
  • PATCH /api/forms/:id/draft/autosave took 85s and returned 409.
  • One public form submit took 36s; another took 282s and ended as client closed/499.
  • Several auth refresh requests logged 499s after 17s-107s.
  • CPU and memory were very low during the window: roughly 0.02 vCPU and ~400 MB RAM on an 8 GB limit.
  • Live health checks were fast before/after the slow cluster.
  • MongoDB Atlas looked healthy from our side: ping/query timings around tens of ms, no active long-running ops, no index builds, small collections.

We also saw lots of long /socket.io/ polling/upgrade requests, which can distort aggregate HTTP percentiles, but the raw REST logs show real slow login/publish/form-submit requests too.

This does not look like sustained app CPU/RAM pressure, database saturation, or normal post-deploy warmup. It feels like intermittent ingress/proxy/network delay or request handling delay outside the obvious app resource metrics.

Railway status currently says operational, so I’m mainly adding this as another datapoint: low app resource usage, healthy DB checks, but sporadic 30s-200s API waits on POST/PATCH/auth flows. Happy to provide timestamps/deployment IDs if helpful.

$10 Bounty

1 Replies

Railway
BOT

3 months ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 3 months ago


ayalaa12
FREETop 5% Contributor

13 days ago

The current evidence rules out sustained CPU/RAM pressure, but it does not yet distinguish a Railway ingress delay from an application process waiting on an I/O dependency. Railway now exposes enough per-request timing to make that distinction without guessing.

Use one slow request ID and build this correlation:

  1. In Railway HTTP Logs, filter the affected paths and slow requests, for example:

    • @path:/api/auth/login @totalDuration:>5000
    • @path:/api/forms/:id/publish @totalDuration:>5000 (use the concrete path shown in the log)

    Record @requestId, @responseTime, @upstreamRqDuration, @totalDuration, @edgeRegion, @rxBytes, status and @responseDetails.

  2. Add a first-line Express middleware before JSON parsing/auth that logs only non-secret metadata: X-Railway-Request-Id, X-Request-Start, X-Railway-Edge, RAILWAY_REPLICA_ID, RAILWAY_REPLICA_REGION, local receive time, and the finish/close time. Railway documents X-Request-Start as the Unix-millisecond time the request was received and the request ID as the HTTP-log correlation key.

  3. Compute Date.now() - Number(req.get('x-request-start')) at that first middleware:

    • A large delta (for example 30-200s) before the first app log, or an HTTP log with no matching app receive log, is evidence of edge/upstream delivery delay.
    • A small delta followed by a long handler duration means the request reached Node promptly; instrument body parsing, auth/session lookup, Mongo connection checkout/query and every outbound HTTP call separately. Low CPU is fully compatible with a blocked socket/pool or an unbounded network wait.
  4. From the same client, compare the Railway-provided domain and the custom domain with a unique harmless request while sending X-Railway-Debug: 1. Capture response headers and timings. X-Railway-Edge identifies the ingress POP and the debug response's X-Railway-Upstream-Zone identifies the deployment zone. If slow samples cluster on one POP/path while the same request is fast through another, include those request IDs in the escalation.

  5. Log RAILWAY_REPLICA_ID on every request. If all slow samples correlate to one replica, restart/replace that replica and inspect its Mongo/socket pools. If they occur before all replicas equally, the evidence points back to the network path. Keep Socket.IO polling separated from REST by path in the log query; its long-lived requests should not be used to explain away slow REST request IDs.

For the app-side branch, add finite timeouts to Atlas/outbound calls and log Mongo pool checkout duration. A 499 only says the client disconnected; the phase timings above identify what it was waiting for.

This produces the exact support packet Railway can act on: UTC timestamp, request ID, edge region, upstream zone, deployment/replica ID, request-start-to-app delta, HTTP-log timings and whether a matching first middleware log exists. It also avoids another blind redeploy.

Official references:


Welcome!

Sign in to your Railway account to join the conversation.

Loading...