2 days ago
Project nucleus-vanilla (9aef30ad-f123-4226-9ad1-0f041642ecb7), env prd (9aa617c3-6c54-4f18-b42b-4c518c0d42ef), service nucleus (7604730d-302f-4327-b2d1-1c3a02a2ed22), US West, 12 replicas, edge jfk1. All times UTC, 2026-07-09. Full evidence doc + HAR files + dashboard screenshot attached.
SUMMARY
After our 16:31 deployment, replicas progressively became unresponsive: two were fully dead for ~56 and ~50 minutes, two more wedged for 5-19 minutes. The whole time, the dashboard showed "12/12 replicas active" (screenshot attached), no restart policy fired, no replica was replaced, and the dead upstreams stayed in edge rotation — so ~25-33% of customer requests hit what looks like a ~5s upstream-timeout+retry penalty (TTFBs quantized at 5.1s / 10.2s / 15.2s, worst cases 42-44s). Recovery required a manual railway restart at ~17:53-17:58.
EVIDENCE (per-replica log silence; every replica ships logs continuously via Datadog, healthy ones log every minute):
60180ce04d2b: last log 16:57:54, silent ~56 min, resumed 17:54 only via manual restart
16fa6d9cdd0f: last log 17:05:22, silent ~50 min, resumed 17:56 only via manual restart (logged an SQS "receipt handle has expired" error 16s before dying — it wedged mid-work, not idle)
10be77ef7b22: silent 17:23:44 → 17:43 (~19 min, self-recovered)
ec18884bed0d: silent ~17:19 → 17:24 (~5 min, self-recovered)
Control: c72319574570 logged in every single minute bucket 16:55→17:59
Distinct actively-logging hosts per minute: 12 (through 16:56) → 11 (16:57) → 10 (17:01) → 9 (17:18-17:42) → 10 (17:43) → 12 only after the manual restart (17:54-17:58). No new hostnames ever appeared between 16:32 and 17:54 — nothing was replaced.
Dashboard contradiction: screenshot captured between ~17:30 and 17:52 shows "12/12 replicas active" / "Deployment successful". Throughout that entire window two of the replicas above were provably silent.
CUSTOMER IMPACT (traceable on your side):
17:19:38 GET /roles/self → 42.5s TTFB then HTTP 304. x-railway-request-id: sS6YFXxrTrujzZlnWUN5dQ (x-hikari-trace: jfk1.48ys)
17:22:57 GET /auth → 44.2s TTFB then HTTP 200. x-railway-request-id: csHTF6ZYR6eCYOKFYqVb7A
Same HAR captures: bare CORS OPTIONS preflights at 5,089-5,151 ms, GETs at 5,212-5,585 ms — fast (~0.15s) and slow requests interleaved same-millisecond on the same HTTP/2 connection, i.e. per-upstream, not network-side.
External unauthenticated probes (curl OPTIONS): 17:26-17:37 saw stalls of exactly 5.1s / 10.2s / 15.2s on ~25-33% of requests, one hang of 4.5+ min, one connection failure. Post-restart (17:59+): consistently 0.13-0.27s, zero stalls.
Platform-wide: our PostHog error tracking shows fetch failures from ≥7 users across ≥7 customer domains during the worst window.
QUESTIONS
Why did the dashboard report 12/12 replicas "active" while 2-3 were unresponsive for up to 56 min? What does "active" measure — container running, or workload responsive?
Why did no restart policy or health mechanism fire? Is there any RUNTIME liveness probing of replicas (beyond deploy-gating healthchecks), or plans for it?
Can the edge (hikari) eject upstreams that persistently hit the ~5s upstream timeout instead of keeping them in rotation? What are the exact timeout/retry semantics? The two x-railway-request-id values above should let you trace this internally.
Please share per-container CPU/memory graphs for 60180ce04d2b, 16fa6d9cdd0f, 10be77ef7b22 around 16:57 / 17:05 / 17:23 UTC — we need to distinguish OOM-kill from event-loop hang to root-cause our application side.
Attachments
3 Replies
2 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 2 days ago
2 days ago
The dashboard shows active based on container state, this doesn't mean the workload itself is responsive.
Health check mechanisms only monitor the container status, not if the internal application event loop is blocked/ hung.
Edge proxy keeps up streams in rotation as long as they respond to liveness checks even with persistent timeouts unless a specific ejection policy triggers.
Can you share the specific metrics graphs for the containers mentioned in the report to distinguish OOM-kill from event-loop?
2 days ago
Thanks — that matches what we found. To answer your question: the metrics point to a memory-driven hang, not an OOM-kill.
Attached is the service memory graph (from railway metrics --raw, 60s samples, UTC). Fleet baseline was ~7 GB across 12 replicas after the 16:31 deploy. At 16:57 — the exact minute the first replica stopped logging — memory begins climbing, plateauing at 23–29 GB until our manual restart at ~17:56, then instantly returns to ~8 GB. The processes never exited: restart policy is ALWAYS and never fired, no boot logs appear until the manual restart, and deployment.instances[].status reported RUNNING for all 12 the whole time. An OOM-kill (exit 137) would have produced a restart; instead the processes stayed alive with the event loop starved — consistent with a V8 GC death spiral as heap approached its ceiling. CPU never saturated (peak 8.7/24 vCPU).
The three affected deploymentInstanceIds, if you can check platform-side: fc656f82-a404-476a-8ee0-069aabc380ea (silent 16:57:54Z), c0f99793-a010-4bec-a0f5-d6f4f917b690 (17:05:22Z), 6a5ecaea-308a-4107-a3d3-128d52bd45c3 (17:23:44Z, self-recovered 17:43Z). Per-replica dashboard graphs attached as well.
Given healthchecks are deploy-gate only and restart policy is exit-only, is there any roadmap for runtime liveness probing or edge ejection of upstreams that persistently hit the dial timeout? That's the gap that turned three sick replicas into an hour of platform-wide latency.
Attachments
an hour ago
Alright. The memory is being caused by heap reaching it's ceiling.
Adjusting the health check configurations, by adding runtime liveness probing can help you catch unresponsive instances more quickly.
You can find the information on their public roadmap at station.railway.com/roadmap
Look into edge ejection for upstreams that keep timing out. Also review overal memory limits and scaling settings to ensure your app has enough headroom