Frequent silent container terminations, one without auto-replace leaving performance degraded
jonbaer-bit
PROOP
4 days ago
We're seeing an elevated rate of container terminations on our production service and one incident where a terminated replica was not replaced. Can you please help us to look into this?
Identifiers
- Project:
marketscale-cms(81a4fbf7-8dc6-4515-88e2-56753674e705) - Environment:
production(2c56f19e-f847-4e9c-9931-497618f3a7c3) - Service:
payload-cms(2ceb67d3-df6e-463f-9ff6-a5b0f4c757af), regionus-east4-eqdc4a, 2 replicas at the time (3 since Jul 3 20:07 UTC)
Main incident: Jul 3, 19:34 UTC
- Deployment
ec65b95c-ac9c-4768-9c9b-c478a82ff38e(2 replicas). At 19:34 UTC one replica was terminated mid-traffic. Its memory was flat (~1.2GB against a much higher limit), and there are no crash indicators in our application logs / no graceful-shutdown output. - No replacement container was scheduled for approx 15 minutes. The deployment was marked crashed (we received the webhook at ~19:36 UTC). Production ran on a single replica until we manually redeployed at 19:40 UTC (new deployment
6acc1b31-8c53-4b12-a3ac-40f958fa3231, healthy at 19:48 UTC).
Background pattern: Jul 2 23:00 UTC onward
- On the same deployment we count ~13 additional silent container terminations at a roughly 1–2 hour cadence (container boot timestamps, UTC, Jul 2–3): 23:29, 23:32, 00:42, 01:47, 03:16, 05:28, 07:10, 09:08, 10:59, 11:53, 12:47, 13:38, 18:34. Those were all replaced within seconds (unlike the 19:34 one). Five similar restarts occurred on Jul 2 during the prior deployment.
- After restarts, our MongoDB connection metadata shows the app containers landing on hosts with different kernels (
6.12.12+bpo-cloud-amd64,6.18.15+deb13,6.19.10+deb13), suggesting host migrations/rebalancing. - Separately, on Jul 2 08:36–08:59 UTC the app containers lost network reachability to our MongoDB service (same project, private network) for 23 minutes while MongoDB itself stayed up, and the timing coincides with a host kernel change on the app containers.
Questions
- What is terminating these containers? Host maintenance, rebalancing, or something else? Can you share the exit reasons/host events your side records for the timestamps above?
- Why was the 19:34 UTC termination not auto-replaced, and what marked the deployment crashed? Does the restart policy behave differently for replica-level terminations?
- Should we expect this churn rate to continue (e.g., is planned maintenance in progress in us-east4)?
- Is any of this related to the US-East TCP proxy degradation your status page shows for July?
We can provide log excerpts for any of the windows above. Thank you!
0 Replies
Status changed to Awaiting Railway Response Railway • 4 days ago
Railway
BOT
3 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 3 days ago