9 days ago
Hi Railway team,
We have a production worker deployment that has been stuck in BUILDING at the build-scheduling stage for over 2.5 hours and has never emitted a build step.
The complete build log is:
scheduling build on Metal builder "builder-lnxcux"
An identical build on the same service and Dockerfile completed end-to-end in roughly 9 minutes earlier today.
The deployment was created by:
railway scale --service worker --environment production us-west2=0
We ran this to verify a scale-to-zero rollback procedure. The CLI reported 0 replicas configured, but Railway implemented the scale operation as a new deployment.
Importantly, the previous worker deployment is still serving normally with one running replica. It has not restarted, received SIGTERM, or cut over, so production remains healthy.
The service is source-less and normally deploys via railway up from CI using:
/apps/api/railway.worker.toml
with:
builder: DOCKERFILE
dockerfilePath: apps/api/Dockerfile
Region: us-west2
Railway CLI: 5.26.1
We have not attempted to clear the stuck deployment because the CLI exposes no explicit deployment-cancel command. railway down is documented as removing the most recent deployment, and we do not want to risk removing the healthy serving deployment.
Could you please clarify:
How can we safely cancel/remove only the stuck BUILDING deployment while leaving the currently serving deployment untouched?
Can Railway clear/resume the stuck deployment on your side?
Is the pending scale ...=0 replica intent stored independently of that deployment? If so, how can we safely inspect/reset it to 1 without affecting the serving deployment?
Is a deployment that remains at scheduling build for this long expected to recover on its own?
This is blocking production maintenance, so we do not want to initiate another production deployment until the state is understood.
Thanks.
4 Replies
9 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 9 days ago
9 days ago
Try removing/aborting the deployment via the web dashboard and redeploy when it’s aborted.
9 days ago
Thanks — aborting from the dashboard worked cleanly, exactly as suggested.
▎
▎ The stuck deployment moved straight to REMOVED, and importantly the healthy serving deployment was left completely untouched: it stayed SUCCESS and active throughout, with no restart, no SIGTERM and no interruption to the running process. Our worker has now been up continuously for over nine hours across the abort, with an unbroken heartbeat counter. The service shows 1 Replica and is Online, and the previously stuck deployment is gone from the active set.
▎
▎ One remaining question before we schedule our next deployment. The stuck deployment never emitted a single build step — its entire build log was scheduling build on Metal builder "builder-lnxcux" for over two and a half hours, against a ~9 minute baseline for the same Dockerfile on the same service earlier that day. Since it was aborted rather than completing, we have no evidence either way about whether the underlying build-scheduling problem is cleared.
▎
▎ Can you confirm whether that issue is resolved, and whether a new deployment for this project and service should now schedule and build normally? We have a change we would rather not deploy blind, so a yes from your side is worth more to us than finding out by trying.
▎
▎ Happy to provide the project/service/deployment identifiers if that helps — I've left them out here since this is a public thread.
▎
▎ Thanks again for the quick steer.
9 days ago
Quick update, and one narrowed question.
Aborting the stuck deployment from the dashboard worked cleanly — it moved to REMOVED and the healthy serving deployment was untouched throughout, still running with no restart. Thanks for that.
Since then we've had a successful deployment on the same project: we used the native Regenerate Password flow on our Postgres service, which redeployed it. That went through normally — new deployment reached SUCCESS, one replica, the database came back and reconnected fine, and our applications were unaffected. Production is healthy.
That tells us deployments on the project are working generally, but it doesn't answer our actual question, because Postgres is deployed from a public image — it's an image pull, not a build. Our stuck deployment was a Dockerfile build, and it never emitted a single build step: the whole log was scheduling build on Metal builder for over two and a half hours, against a ~9 minute baseline for that same Dockerfile on that same service earlier the same day.
So, specifically: should Dockerfile builds on this project now schedule and build normally? Is the earlier build-scheduling problem understood or resolved on your side, and was there anything particular about that builder?
We have a change ready to deploy and would rather have a yes from your side than find out by trying. Happy to share the project/service/deployment identifiers if useful — leaving them out here since this is public.
Thanks again.
9 days ago
can u share a picture