a month ago
Service: select api-worker schedule runner api (ID 2532ef5a-a4a5-4ef0-bfaf-b899fc25122c)
Visibility: if you have a private/priority support option on your plan, use it — this is an infra question better handled privately. Otherwise "Ask the Community" is fine.
Description:
Project: gentle-intuition (3d9fc731-2f09-4a71-8c33-3d942d4a4701)
Service: api-worker schedule runner api (2532ef5a-a4a5-4ef0-bfaf-b899fc25122c)
Environment: production (773f8b2b-19c0-4f2e-b923-4c8eefa91fd3) — Region: EU West
Current deployment: fea096f7-99f0-48c2-98d3-6df4d1515fce
Also seen recycling: 1aa6c50a-20cf-4de3-af7f-743ef67becc4, 61afccc2-2f46-478d-86fb-51efaffe6ccf
ISSUE
This service's container is restarted repeatedly ON THE SAME DEPLOYMENT (no redeploy) at irregular ~9–40 minute intervals. It's a Node/Bull background worker (cron scheduler + queue consumer), NOT a request server — inbound traffic is ~0.
PROVEN NOT TO BE THE APPLICATION (in-process instrumentation across many cycles):
- Idle at time of kill: CPU ~0% (peak 0.14 vCPU / 24), memory flat (peak 190 MB / 24 GB, ~1%), ingress ~0 (max 0.08 MB), egress 0.
- No exit event before any recycle: process.on('exit') and 'beforeExit' never fire; the SIGTERM/SIGINT handler never logs. The process is hard-killed (SIGKILL) with no grace signal — nothing a JS hook can catch.
- Not memory/OOM, not CPU, not an app crash (unhandled errors are trapped and logged), and NOT the healthcheck (we cleared the Healthcheck Path — it still recycled after ~35 min).
- Recycle intervals ~9 / 18 / 35 / 40 min are non-periodic and match no cron cadence; no CPU/memory/traffic spike precedes any kill. So the workload is not triggering the restarts.
QUESTIONS
- What restart reason / exit code do you record for these recycle events on this container/deployment?
- Is this container subject to node maintenance, bin-packing/rebalancing, preemption, or a quota / noisy-neighbor flag?
- Can you share the container/node lifecycle event log for this service so we can see why it's being recycled?
We've made the workload restart-tolerant on our side, so this is not an outage — but a healthy, idle, traffic-less worker being SIGKILLed roughly every ~15 min shouldn't be normal, and we'd like to understand and stop it.
1 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
19 days ago
Two reframes that should narrow this down fast:
1. The restarts are your restart policy doing its job — the kill itself is the event to explain
Railway's default restart policy for services is Always: "Railway will automatically restart your service every time it stops, regardless of the reason" (docs, Restart policy). So a container that gets killed outside the app (no SIGTERM/SIGINT, nothing catchable in JS) is exactly what you'd see: same deployment, repeated restarts, no redeploy. The restart policy hides the kill; it doesn't cause it.
2. Nothing in Railway's docs recycles services for idleness — so it's not an idle timeout
There is no documented scale-to-zero/idle-recycling behavior for services. Combined with your non-periodic 9/18/35/40-min intervals and zero preceding resource/traffic activity, the most consistent explanation is node-level events: host maintenance or capacity rebalancing/migration moving your container between nodes. That produces precisely your signature — same deployment ID, hard kill, irregular cadence, healthy idle workload. It also explains why clearing the healthcheck changed nothing.
3. What to ask Railway (this is the part only they can answer)
You're on PRO — use the private priority support channel you hinted at, and ask for the container/node lifecycle event log for deployment fea096f7 (plus the two recycling ones):
- The recorded restart reason / exit code for each recycle event.
- Whether host maintenance or rebalancing/migration ran in those windows (they can correlate node IDs).
- Whether there are scheduled maintenance windows you can plan around.
That's the definitive path — community members can't see instance lifecycle events; Railway is the only one holding them.
4. Hardening while you wait (worth doing regardless of root cause)
- BullMQ stalled-job watchdog is your friend:
stalledInterval(default 30s) +maxStalledCountre-queues jobs whose worker died mid-processing — keepattempts+backoffset and handlers idempotent so a SIGKILL mid-job is at-least-once, never lost. - Repeatable jobs live in Redis — when the scheduler comes back, BullMQ re-evaluates the cron state, so a kill doesn't delete your schedule; it can delay a tick while the process is down. Design the consumer to tolerate late/duplicate ticks.
- Move pure scheduling off the process: Railway has a native Cron jobs feature — using it for the "fire at 9:00" part means the SIGKILLed process only has to be a resilient queue consumer, not a scheduler that must be alive at trigger time.
- External heartbeat: a cron or uptime probe hitting a
/health-style endpoint every minute gives you an independent kill-cadence record to correlate against Railway's status page and support's answers.
TL;DR: Restart policy "Always" is auto-restarting a container that's being hard-killed at the node level (maintenance/rebalancing) — not an idle timeout, not your app. Open the private priority support thread with the deployment IDs and ask for the instance lifecycle events/restart reasons; meanwhile, lean on BullMQ's stalled-job watchdog, idempotent handlers, and Railway Cron for scheduling so a kill is invisible to your workload.