Successful historical redeployments automatically removed while service latest remains stopped
conorbmcc03
PROOP

19 days ago

I’m seeing a production service get removed by Railway after a successful historical redeploy, even though the application is healthy immediately beforehand.

Service: superb-adaptation

I’ve reproduced this three times. Each redeploy starts successfully, passes health/readiness checks, continues serving traffic, and shows no crash loop, fatal exception, OOM, or application-level failure. After roughly 40–45 minutes, Railway logs Stopping Container and the deployment becomes REMOVED.

Example incident:

Deployment starts successfully

Health/readiness checks pass

Application collectors continue running successfully

Memory remains normal

No OOM / crash / restart loop

Strict V3 research health remains fresh immediately before removal

Railway then stops the container externally and marks the deployment removed

The redeploys are historical redeployments of a known-good image:

sha256:94382c9c569273097f9bb0d22e331cb5c33c55cdd1d372b1e8c42905f7865ff7

The service’s canonical latestDeployment appears to still reference an older stopped/failed deployment rather than the healthy historical redeploy.

I have checked the obvious client-side causes and found:

no Railway CLI process issuing destructive commands

no cron / at jobs

no local cleanup watcher

no GitHub workflow capable of mutating Railway

no active EventEdge orchestrator capable of removing deployments

Railway auto-deploy is disabled

no application crash or OOM around the removal

I’d appreciate help understanding:

Can Railway lifecycle reconciliation remove a healthy historical deploymentRedeploy if the service’s canonical latestDeployment still points to an older failed/stopped deployment?

Is there a supported way to make a known-good historical redeploy become the stable canonical active deployment without rebuilding current repository HEAD?

Is there any Railway-side lifecycle rule that would explain a healthy deployment being removed after ~40–45 minutes?

I’ve removed internal project/environment/service/audit UUIDs from this public post. I can provide those privately to Railway staff if needed.

Thanks.

$20 Bounty

3 Replies

Status changed to Awaiting Railway Response Railway • 19 days ago


Railway
BOT

19 days ago

This is the kind of problem the Railway community is well placed to help with, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public and the thread closes.

Status changed to Awaiting User Response Railway • 19 days ago


Railway
BOT

19 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 19 days ago


bl4ckph4ntom1
PROTop 10% Contributor

19 days ago

That sounds a bit odd to me, especially since the new redeploy is actually healthy before Railway removes it.

The first thing I'd check is whether you have any deployment overlap/teardown time configured around 40–45 minutes.

Could be one of these:

RAILWAY_DEPLOYMENT_OVERLAP_SECONDS

Service Settings → Deployment Teardown → Overlap

or deploy.overlapSeconds in railway.toml / railway.json.

Railway normally only keeps one deployment active per service, so when a new one becomes active the previous one eventually gets removed. If Railway is somehow treating this healthy historical redeploy as the "previous" one, that could explain why it disappears after roughly the same amount of time every time.

If overlap is 0 or not set though, and the redeploy is still SUCCESS while serviceInstance.latestDeployment.id points to the older failed/stopped deployment, then I'd suspect the service state is getting out of sync somewhere rather than this being normal cleanup.

For getting the known-good version back into the active slot, I'd also try Rollback instead of Redeploy if that deployment still has canRollback: true.

mutation {
  deploymentRollback(id: "<known-good-deployment-id>") {
    id
    status
  }
}

That should restore the old image/variables as a fresh active deployment without rebuilding it.

If Rollback isn't available anymore because the image is outside the retention window, then Redeploy is the fallback.

If none of the overlap settings line up with the ~40–45 minute timing, I'd send Railway the old latestDeployment ID, the new successful redeploy ID, and the timestamps for when it went SUCCESS and when it was removed.

That would give them something pretty concrete to trace on their side.


bl4ckph4ntom1

That sounds a bit odd to me, especially since the new redeploy is actually healthy before Railway removes it. The first thing I'd check is whether you have any deployment overlap/teardown time configured around 40–45 minutes. Could be one of these: `RAILWAY_DEPLOYMENT_OVERLAP_SECONDS` Service Settings → Deployment Teardown → Overlap or `deploy.overlapSeconds` in `railway.toml` / `railway.json`. Railway normally only keeps one deployment active per service, so when a new one becomes active the previous one eventually gets removed. If Railway is somehow treating this healthy historical redeploy as the "previous" one, that could explain why it disappears after roughly the same amount of time every time. If overlap is `0` or not set though, and the redeploy is still `SUCCESS` while `serviceInstance.latestDeployment.id` points to the older failed/stopped deployment, then I'd suspect the service state is getting out of sync somewhere rather than this being normal cleanup. For getting the known-good version back into the active slot, I'd also try Rollback instead of Redeploy if that deployment still has `canRollback: true`. ```graphql mutation { deploymentRollback(id: "<known-good-deployment-id>") { id status } } ``` That should restore the old image/variables as a fresh active deployment without rebuilding it. If Rollback isn't available anymore because the image is outside the retention window, then Redeploy is the fallback. If none of the overlap settings line up with the ~40–45 minute timing, I'd send Railway the old `latestDeployment` ID, the new successful redeploy ID, and the timestamps for when it went SUCCESS and when it was removed. That would give them something pretty concrete to trace on their side.

conorbmcc03
PROOP

18 days ago

Thanks — I checked the overlap/draining configuration.

overlapSeconds = null

drainingSeconds = null

no RAILWAY_DEPLOYMENT_OVERLAP_SECONDS

no RAILWAY_DEPLOYMENT_DRAINING_SECONDS

no config-as-code override

So the repeated ~35–45 minute removals are not explained by deployment overlap.

One service-state anomaly remains:

latestDeployment still points to failed deployment

d0098633-a6ab-4731-8309-f17820b9f037

that deployment is from September 10

it is older than multiple later healthy deployments that Railway subsequently removed

most recent normal GitHub-backed deployment:

2486bd58-5b33-4b01-aec7-f9a448ce8fbb

related removal audit event:

019e0f4c-459c-4502-8fad-2665014cc18d

I also already tried both historical Redeploy and Rollback, and a normal GitHub-backed deployment from a dedicated production branch. All were healthy and then removed.

Could you or a Railway employee confirm whether the stale latestDeployment pointer / service-state reconciliation could be causing this?

If so, what is the supported way to make a healthy deployment become the canonical current deployment?

I’m holding off on further deployments until I know the correct fix.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...