Orphaned deployment container still running, cannot be killed via API or dashboard
zorak2d
HOBBYOP

6 days ago

I have a critical issue: a deployment that Railway marks as "REMOVED" in the dashboard still has a live container running in Google Cloud (us-east4), and I cannot kill it through any available means.

The orphaned container was accessing my database with stale credentials from June 26. I mitigated this by rotating all database API keys on June 29 — the zombie can no longer read or write to my database (all requests return 401). However, the container is still running my full application stack with other credentials from June 26, and I cannot kill it.

--- KEY DETAILS ---

Orphaned deployment: e643391f-3869-4d6f-8ea5-5e992f73a606

  • Created: 2026-06-26T23:57:32Z
  • Status (API): REMOVING / Status (Dashboard): REMOVED
  • Region: us-east4-eqdc4a (GCP)
  • Image digest: sha256:189196042f779dbf94b1b8b2a150d4107e582c9d5975b268b82ab33190f096a9

Current healthy deployment: 9857e0c3-6870-449c-9001-4ed0882f97ee (SUCCESS)

--- HOW I DISCOVERED IT ---

I saw recurring "Legacy API keys are disabled" errors in my Railway logs every ~30 seconds, even though my current deployment has the correct new keys. I traced the errors via deployment ID in the log stream and found they come from deployment e643391f (June 26), not my current deployment 9857e0c3.

Database edge logs confirmed the failing requests come from a different IP:

  • Current deployment: 44.220.117.213 (Amazon AWS) — new keys, working
  • Orphaned container: 35.186.160.222 (Google LLC) — legacy JWT keys, 401

SSH into the current container confirmed it has the correct new keys. No hardcoded keys anywhere in the codebase.

API query confirmed the zombie is still alive:

query { deployment(id: "e643391f-...") { status } }

Response: "REMOVING" — not REMOVED.

--- WHY THIS IS RAILWAY INFRASTRUCTURE ---

  1. The IP 35.186.160.222 is Google LLC. The deployment metadata shows region us-east4-eqdc4a (GCP). My repo is private and I don't use Google Cloud — Railway is my only infrastructure.
  2. The orphaned container logs to my Railway service's log stream — only Railway-managed containers can do this.
  3. It was built from my master branch — my code, deployed by Railway, never properly terminated.

--- WHAT I'VE TRIED (ALL UNSUCCESSFUL) ---

deploymentRemove — returns true, no effect

deploymentCancel — returns true, no effect

deploymentRestart — error: "Deployment is not restartable"

Redeploy from dashboard — would replace my healthy deployment with stale code, not viable

20+ service redeploys since June 26 — the orphan survived all of them

--- SECURITY TIMELINE ---

June 26: Zombie deployment created in us-east4 (GCP) with legacy database keys.

June 26–29: 3 days of unauthorized service_role read/write access to my production database. It caused data corruption I had to manually repair. No audit trail of what was read/written.

June 29: I rotated all database keys. Zombie's DB access blocked (401). Access mitigated.

June 29–July 1: Zombie still running, making failed DB requests every 30s, holding other credentials from June 26.

July 1: Discovered the zombie via deployment ID tracing. All API kill attempts failed.

--- IMPACT ---

  1. Unauthorized DB access (June 26–29): 3 days of full service_role access — bypasses all RLS. Caused data corruption, manually repaired. Mitigated by key rotation on June 29.
  2. Duplicate application instance: runs my full gateway with credentials from June 26, may be consuming paid external APIs independently.
  3. Queue contention: caused 5 DB records incorrectly marked as failed, manually reset.
  4. Rate limit conflicts: both instances hit external APIs with the same credentials.
  5. Stale code: the zombie runs code from June 26, missing critical bug fixes deployed since then.

--- WHAT I NEED ---

Force-kill the orphaned container in region us-east4-eqdc4a for deployment e643391f-3869-4d6f-8ea5-5e992f73a606 (image digest: sha256:189196042f779dbf94b1b8b2a150d4107e582c9d5975b268b82ab33190f096a9).

Thank you for your help.

Best regards

Solved

1 Replies

Status changed to Awaiting Railway Response Railway 6 days ago


6 days ago

We have stopped the orphaned deployment.


Status changed to Awaiting User Response Railway 6 days ago


Status changed to Solved zorak2d 6 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...