Hey! I searched across our infrastructure for that container (613b964be9c3) but wasn't able to locate it on the hosts I checked.

A couple of questions to help track this down: is the duplicate beat behavior still happening right now? If so, could you share a recent log snippet showing it? Also, how did you identify that container hostname and IP - was it from your celery logs, Redis CLIENT LIST, or somewhere else?

If you're able to run CLIENT LIST on your Redis and that IP (fd12:ac45:f77c:0:a000:39:8689:5594) still shows as connected, the container is definitely still running and I can dig deeper. If it's gone, the orphan may have already been cleaned up.

Yes, the duplicate beat is still actively happening right now.

Here's a fresh log snippet from our celery workers showing health_check (scheduled every 2 minutes) being executed twice — once from our legitimate beat and once from the ghost:

14:16:47 health_check[0dd1de1d...] — ghost beat (:47 offset)

14:17:03 health_check[1b75bee0...] — our beat (:03 offset)

14:18:47 health_check[03d87d22...] — ghost beat

14:19:03 health_check[a21e41cf...] — our beat

Two different task IDs, ~16 seconds apart.

The ghost IP is still connected to Redis right now.

Here's the CLIENT LIST output showing its two active connections:

id=4 addr=[fd12:ac45:f77c:0:a000:39:8689:5594]:45538 age=41147s (11.4 hours) idle=46s cmd=lpush

id=269 addr=[fd12:ac45:f77c:0:a000:39:8689:5594]:44426 age=40188s (11.2 hours) idle=46s cmd=unsubscribe sub=1

These connection ages (11+ hours) correspond to a Redis restart we did last night. The container reconnected immediately after the restart — it has valid Redis credentials and auto-reconnects when killed.

How we identified it:

We traced the container hostname 613b964be9c3 from Celery error logs on the worker side. When the ghost beat dispatches drop_check_ready_batches (a task the current workers don't have registered), the worker logs the full task headers including 'origin': 'gen1@613b964be9c3'. The gen1@ prefix is Celery beat's process identifier. We then correlated the ghost's dispatch timing with CLIENT LIST to identify the IP. We've tried CLIENT KILL multiple times — it reconnects within seconds. We also redeployed all three app services (backend, celery-workers, celery-beat) and restarted Redis — the ghost survived all of it.

Good news - I tracked down the ghost container. It was an orphaned deployment instance (09a3f174-03a4-49bb-890f-4cf6f2c48dc8) from April 6 that was running on production-stacker-dc4a-metal-016-03. It should have been terminated when newer deployments came up, but the cleanup never happened - it kept running with valid Redis credentials for 10 days.

The container stopped about 4 hours ago and is no longer present on the stacker. Can you check if the duplicate beats have stopped? You should see only one health_check task per 2-minute interval now, and CLIENT LIST on your Redis should no longer show connections from fd12:ac45:f77c:0:a000:39:8689:5594.

If the duplicates are still happening, let me know and I'll dig deeper - but based on our metrics the orphan appears to be gone.