6 days ago
Hi,
A container with hostname ec4d4f402716 is still running old code and causing production errors, despite multiple redeploys and restarts.
Project: af910c21-908e-49b3-a473-98ec533fad2b
Environment: development
Service: backend (ad0962cb-88a5-4361-b94d-2981aaebe99d)
Evidence:
Active deployment: 60e1cac5-cabf-457f-bd0e-c6f9a4937b8d (created 2026-07-01 05:28:26)
Zombie hostname: ec4d4f402716 (Bun 1.3.11)
Current active hostname: 05e2f7745d9d (Bun 1.3.14)
Fresh error from zombie at 2026-07-01T06:22:13Z (after redeploy completed)
Errors: Postgres queries to dropped table email_campaigns, Redis ETIMEDOUT (stale networking)
Attempted fixes:
Multiple redeploys
Service restart
Added drain/overlap settings (drainingSeconds: 30, overlapSeconds: 10)
Scaling replicas (1→2→1)
The zombie container is not visible in the Deployments list but is actively serving requests with stale code. This requires cluster-level investigation.
Thank you so much
Thomas,
1 Replies
6 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 6 days ago
6 days ago
Based on the evidence, this doesn't look like an application issue—it looks like an instance lifecycle or service-discovery problem.
The key indicators are:
- The active deployment is
60e1cac5-cabf-457f-bd0e-c6f9a4937b8d. - Requests are still being handled by hostname
ec4d4f402716, which is running an older Bun version (1.3.11). - That hostname is no longer visible in the Deployments list but continues serving traffic after multiple redeploys and restarts.
This suggests the old container may not have been fully deregistered from the routing layer or is still attached to the service endpoint.
Things I'd check on the platform side:
- Verify that
ec4d4f402716still exists on any node in the cluster. - Check whether it is still registered in the service-discovery/load-balancer backend despite the deployment being marked inactive.
- Compare the routing table/backend targets with the deployment state to ensure only containers from deployment
60e1cac5-cabf-457f-bd0e-c6f9a4937b8dare receiving traffic. - Verify that connection draining completed successfully and that the old instance was removed after
drainingSecondselapsed. - Review scheduler/orchestrator logs around the deployment window to determine whether container termination failed or the deregistration event was missed.
One additional diagnostic that may help confirm the issue is to have each container expose its deployment ID, hostname, and build version through a simple health endpoint or response header. If requests alternate between 05e2f7745d9d and ec4d4f402716, that would confirm the router is still forwarding traffic to an orphaned instance.
Given the evidence, my leading hypothesis is an orphaned backend entry in the routing/service-discovery layer rather than a failed application deployment.