3 months ago
We’re seeing what looks like the same pattern on our Railway-hosted Express/Node backend.
Region/platform: Railway app service with MongoDB Atlas backend
Started noticing: after our June 3 deploy, but symptoms continued well beyond normal warmup.
Symptoms:
- Intermittent very slow API responses, especially authenticated writes/actions.
POST /api/auth/logintook 15s and later 42s.POST /api/forms/:id/publishtook 42s.PATCH /api/forms/:id/draft/autosavetook 85s and returned 409.- One public form submit took 36s; another took 282s and ended as client closed/499.
- Several auth refresh requests logged 499s after 17s-107s.
- CPU and memory were very low during the window: roughly 0.02 vCPU and ~400 MB RAM on an 8 GB limit.
- Live health checks were fast before/after the slow cluster.
- MongoDB Atlas looked healthy from our side: ping/query timings around tens of ms, no active long-running ops, no index builds, small collections.
We also saw lots of long /socket.io/ polling/upgrade requests, which can distort aggregate HTTP percentiles, but the raw REST logs show real slow login/publish/form-submit requests too.
This does not look like sustained app CPU/RAM pressure, database saturation, or normal post-deploy warmup. It feels like intermittent ingress/proxy/network delay or request handling delay outside the obvious app resource metrics.
Railway status currently says operational, so I’m mainly adding this as another datapoint: low app resource usage, healthy DB checks, but sporadic 30s-200s API waits on POST/PATCH/auth flows. Happy to provide timestamps/deployment IDs if helpful.
1 Replies
3 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 3 months ago
13 days ago
The current evidence rules out sustained CPU/RAM pressure, but it does not yet distinguish a Railway ingress delay from an application process waiting on an I/O dependency. Railway now exposes enough per-request timing to make that distinction without guessing.
Use one slow request ID and build this correlation:
-
In Railway HTTP Logs, filter the affected paths and slow requests, for example:
@path:/api/auth/login @totalDuration:>5000@path:/api/forms/:id/publish @totalDuration:>5000(use the concrete path shown in the log)
Record
@requestId,@responseTime,@upstreamRqDuration,@totalDuration,@edgeRegion,@rxBytes, status and@responseDetails. -
Add a first-line Express middleware before JSON parsing/auth that logs only non-secret metadata:
X-Railway-Request-Id,X-Request-Start,X-Railway-Edge,RAILWAY_REPLICA_ID,RAILWAY_REPLICA_REGION, local receive time, and thefinish/closetime. Railway documentsX-Request-Startas the Unix-millisecond time the request was received and the request ID as the HTTP-log correlation key. -
Compute
Date.now() - Number(req.get('x-request-start'))at that first middleware:- A large delta (for example 30-200s) before the first app log, or an HTTP log with no matching app receive log, is evidence of edge/upstream delivery delay.
- A small delta followed by a long handler duration means the request reached Node promptly; instrument body parsing, auth/session lookup, Mongo connection checkout/query and every outbound HTTP call separately. Low CPU is fully compatible with a blocked socket/pool or an unbounded network wait.
-
From the same client, compare the Railway-provided domain and the custom domain with a unique harmless request while sending
X-Railway-Debug: 1. Capture response headers and timings.X-Railway-Edgeidentifies the ingress POP and the debug response'sX-Railway-Upstream-Zoneidentifies the deployment zone. If slow samples cluster on one POP/path while the same request is fast through another, include those request IDs in the escalation. -
Log
RAILWAY_REPLICA_IDon every request. If all slow samples correlate to one replica, restart/replace that replica and inspect its Mongo/socket pools. If they occur before all replicas equally, the evidence points back to the network path. Keep Socket.IO polling separated from REST by path in the log query; its long-lived requests should not be used to explain away slow REST request IDs.
For the app-side branch, add finite timeouts to Atlas/outbound calls and log Mongo pool checkout duration. A 499 only says the client disconnected; the phase timings above identify what it was waiting for.
This produces the exact support packet Railway can act on: UTC timestamp, request ID, edge region, upstream zone, deployment/replica ID, request-start-to-app delta, HTTP-log timings and whether a matching first middleware log exists. It also avoids another blind redeploy.
Official references: