19 days ago
I’m seeing a production service get removed by Railway after a successful historical redeploy, even though the application is healthy immediately beforehand.
Service: superb-adaptation
I’ve reproduced this three times. Each redeploy starts successfully, passes health/readiness checks, continues serving traffic, and shows no crash loop, fatal exception, OOM, or application-level failure. After roughly 40–45 minutes, Railway logs Stopping Container and the deployment becomes REMOVED.
Example incident:
Deployment starts successfully
Health/readiness checks pass
Application collectors continue running successfully
Memory remains normal
No OOM / crash / restart loop
Strict V3 research health remains fresh immediately before removal
Railway then stops the container externally and marks the deployment removed
The redeploys are historical redeployments of a known-good image:
sha256:94382c9c569273097f9bb0d22e331cb5c33c55cdd1d372b1e8c42905f7865ff7
The service’s canonical latestDeployment appears to still reference an older stopped/failed deployment rather than the healthy historical redeploy.
I have checked the obvious client-side causes and found:
no Railway CLI process issuing destructive commands
no cron / at jobs
no local cleanup watcher
no GitHub workflow capable of mutating Railway
no active EventEdge orchestrator capable of removing deployments
Railway auto-deploy is disabled
no application crash or OOM around the removal
I’d appreciate help understanding:
Can Railway lifecycle reconciliation remove a healthy historical deploymentRedeploy if the service’s canonical latestDeployment still points to an older failed/stopped deployment?
Is there a supported way to make a known-good historical redeploy become the stable canonical active deployment without rebuilding current repository HEAD?
Is there any Railway-side lifecycle rule that would explain a healthy deployment being removed after ~40–45 minutes?
I’ve removed internal project/environment/service/audit UUIDs from this public post. I can provide those privately to Railway staff if needed.
Thanks.
3 Replies
Status changed to Awaiting Railway Response Railway • 19 days ago
19 days ago
This is the kind of problem the Railway community is well placed to help with, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public and the thread closes.
Status changed to Awaiting User Response Railway • 19 days ago
19 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 19 days ago
19 days ago
That sounds a bit odd to me, especially since the new redeploy is actually healthy before Railway removes it.
The first thing I'd check is whether you have any deployment overlap/teardown time configured around 40–45 minutes.
Could be one of these:
RAILWAY_DEPLOYMENT_OVERLAP_SECONDS
Service Settings → Deployment Teardown → Overlap
or deploy.overlapSeconds in railway.toml / railway.json.
Railway normally only keeps one deployment active per service, so when a new one becomes active the previous one eventually gets removed. If Railway is somehow treating this healthy historical redeploy as the "previous" one, that could explain why it disappears after roughly the same amount of time every time.
If overlap is 0 or not set though, and the redeploy is still SUCCESS while serviceInstance.latestDeployment.id points to the older failed/stopped deployment, then I'd suspect the service state is getting out of sync somewhere rather than this being normal cleanup.
For getting the known-good version back into the active slot, I'd also try Rollback instead of Redeploy if that deployment still has canRollback: true.
mutation {
deploymentRollback(id: "<known-good-deployment-id>") {
id
status
}
}That should restore the old image/variables as a fresh active deployment without rebuilding it.
If Rollback isn't available anymore because the image is outside the retention window, then Redeploy is the fallback.
If none of the overlap settings line up with the ~40–45 minute timing, I'd send Railway the old latestDeployment ID, the new successful redeploy ID, and the timestamps for when it went SUCCESS and when it was removed.
That would give them something pretty concrete to trace on their side.
bl4ckph4ntom1
That sounds a bit odd to me, especially since the new redeploy is actually healthy before Railway removes it. The first thing I'd check is whether you have any deployment overlap/teardown time configured around 40–45 minutes. Could be one of these: `RAILWAY_DEPLOYMENT_OVERLAP_SECONDS` Service Settings → Deployment Teardown → Overlap or `deploy.overlapSeconds` in `railway.toml` / `railway.json`. Railway normally only keeps one deployment active per service, so when a new one becomes active the previous one eventually gets removed. If Railway is somehow treating this healthy historical redeploy as the "previous" one, that could explain why it disappears after roughly the same amount of time every time. If overlap is `0` or not set though, and the redeploy is still `SUCCESS` while `serviceInstance.latestDeployment.id` points to the older failed/stopped deployment, then I'd suspect the service state is getting out of sync somewhere rather than this being normal cleanup. For getting the known-good version back into the active slot, I'd also try Rollback instead of Redeploy if that deployment still has `canRollback: true`. ```graphql mutation { deploymentRollback(id: "<known-good-deployment-id>") { id status } } ``` That should restore the old image/variables as a fresh active deployment without rebuilding it. If Rollback isn't available anymore because the image is outside the retention window, then Redeploy is the fallback. If none of the overlap settings line up with the ~40–45 minute timing, I'd send Railway the old `latestDeployment` ID, the new successful redeploy ID, and the timestamps for when it went SUCCESS and when it was removed. That would give them something pretty concrete to trace on their side.
18 days ago
Thanks — I checked the overlap/draining configuration.
overlapSeconds = null
drainingSeconds = null
no RAILWAY_DEPLOYMENT_OVERLAP_SECONDS
no RAILWAY_DEPLOYMENT_DRAINING_SECONDS
no config-as-code override
So the repeated ~35–45 minute removals are not explained by deployment overlap.
One service-state anomaly remains:
latestDeployment still points to failed deployment
d0098633-a6ab-4731-8309-f17820b9f037
that deployment is from September 10
it is older than multiple later healthy deployments that Railway subsequently removed
most recent normal GitHub-backed deployment:
2486bd58-5b33-4b01-aec7-f9a448ce8fbb
related removal audit event:
019e0f4c-459c-4502-8fad-2665014cc18d
I also already tried both historical Redeploy and Rollback, and a normal GitHub-backed deployment from a dedicated production branch. All were healthy and then removed.
Could you or a Railway employee confirm whether the stale latestDeployment pointer / service-state reconciliation could be causing this?
If so, what is the supported way to make a healthy deployment become the canonical current deployment?
I’m holding off on further deployments until I know the correct fix.