The dashboard shows active based on container state, this doesn't mean the workload itself is responsive.

Health check mechanisms only monitor the container status, not if the internal application event loop is blocked/ hung.

Edge proxy keeps up streams in rotation as long as they respond to liveness checks even with persistent timeouts unless a specific ejection policy triggers.

Can you share the specific metrics graphs for the containers mentioned in the report to distinguish OOM-kill from event-loop?

Thanks — that matches what we found. To answer your question: the metrics point to a memory-driven hang, not an OOM-kill.

Attached is the service memory graph (from railway metrics --raw, 60s samples, UTC). Fleet baseline was ~7 GB across 12 replicas after the 16:31 deploy. At 16:57 — the exact minute the first replica stopped logging — memory begins climbing, plateauing at 23–29 GB until our manual restart at ~17:56, then instantly returns to ~8 GB. The processes never exited: restart policy is ALWAYS and never fired, no boot logs appear until the manual restart, and deployment.instances[].status reported RUNNING for all 12 the whole time. An OOM-kill (exit 137) would have produced a restart; instead the processes stayed alive with the event loop starved — consistent with a V8 GC death spiral as heap approached its ceiling. CPU never saturated (peak 8.7/24 vCPU).

The three affected deploymentInstanceIds, if you can check platform-side: fc656f82-a404-476a-8ee0-069aabc380ea (silent 16:57:54Z), c0f99793-a010-4bec-a0f5-d6f4f917b690 (17:05:22Z), 6a5ecaea-308a-4107-a3d3-128d52bd45c3 (17:23:44Z, self-recovered 17:43Z). Per-replica dashboard graphs attached as well.

Given healthchecks are deploy-gate only and restart policy is exit-only, is there any roadmap for runtime liveness probing or edge ejection of upstreams that persistently hit the dial timeout? That's the gap that turned three sick replicas into an hour of platform-wide latency.

Alright. The memory is being caused by heap reaching it's ceiling.

Adjusting the health check configurations, by adding runtime liveness probing can help you catch unresponsive instances more quickly.

You can find the information on their public roadmap at station.railway.com/roadmap

Look into edge ejection for upstreams that keep timing out. Also review overal memory limits and scaling settings to ensure your app has enough headroom

What's your current status.?

Thanks, heap ceiling matches our diagnosis (it's in our earlier posts). But I've gone through the full public roadmap and neither runtime liveness probing nor upstream ejection appears on it as far as we can tell, neither is configurable today (healthchecks only gate deploys, restart policy only fires on exit). If I've missed something, a link to the specific config would be great.

Meanwhile we've shipped our own in-process watchdog (unresponsive replicas self-terminate so the restart policy can heal them). Happy to share the approach. Still hoping for staff input on the four questions in the original post, especially the per-container memory graphs for the incident window.

For a V8 GC death spiral, that in-process watchdog is highly effective workaround, forcing the unresponsive replica to self-terminate.

The root cause is a memory leak during your 16:31 deployment

The node.js is stuck thrashing the GC to free memory. Use Node.js debugging tools (like heapdump or --inspect flag) in a staging environment to profile the memory and find the leak. I'd also suggest to increase --max-old-space-size temporarily to have more breathing room for heap

Any updates?

Thanks! The watchdog is indeed holding. The deploy-leak theory doesn't fit our data though: yesterday we had zero deploys and still saw 160 involuntary replica boots, clustering exactly on one tenant's traffic peak. It's load-triggered heap ballooning, not a release regression. We already run --max-old-space-size=4096; and we'll profile in production with a continuous profiler since staging can't reproduce the traffic shape. Still hoping for a staff answer on the per-replica restart API / edge ejection / what "N/N active" asserts.

Have you optimized the specific garbage collection flags or tuning options in node.js?