a month ago
COLIB'RI — service web (ad0d5dd1-c8cb-4666-b37d-c32dd1a507ed)
Multiple consecutive deployments on this service get stuck at the PRE_DEPLOY_COMMAND step, which stays in "pending" state indefinitely (observed for 30–60+ minutes with no progress and no failure).
Deploy logs show the pre-deploy command's actual content (migrations, fixtures, etc.) executing and completing successfully — the process finishes normally. However, the deployment status in the dashboard never transitions past "Building" / pre-deploy "pending", and the deployment is never promoted to Active.
Affected deployments (all showing the same symptom):
95f23b02-c982-480e-b5f5-60413a10de02
13d6da40-de73-4003-b2de-7b0fc9568334
6899f15b-39b2-49d0-914a-92fb3d207aae
What we've ruled out on our side:
Healthcheck path is empty (not a healthcheck-related failure)
No locked/idle-in-transaction queries on Postgres (pg_stat_activity clean)
Connection count well under max_connections (7/100)
Build phase completes quickly and successfully every time
The pre-deploy command's SQL/Django operations are simple and idempotent (ALTER TABLE IF NOT EXISTS, migrate, loaddata) — nothing that should hang
This looks like a control-plane issue where the completion of the pre-deploy command isn't being recognized (possibly a missed completion webhook/callback), causing the deployment to remain stuck rather than transitioning to Active or failing outright.
Request: Please investigate why PRE_DEPLOY_COMMAND completion isn't being detected for this service, and/or manually unstick/promote deployment 6899f15b-39b2-49d0-914a-92fb3d207aae if the underlying command actually completed.
1 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 30 days ago
19 days ago
How pre-deploy actually runs (from Railway docs)
Pre-deploy executes in a separate container between build and deploy: it has your app env vars and private-network access, but volumes are not mounted and filesystem changes are not persisted. Two documented constraints matter here:
- "If your command fails, it will not be retried and the deployment will not proceed."
- It "will occupy a slot in your build queue."
1. Check that the process group actually exits (the common local culprit)
The deployment phase doesn't finish when your DB work finishes — it finishes when the command process exits. If anything keeps the shell alive after the migrations complete, you get exactly what you see: logs show the SQL done, status stays pending for 30–60+ minutes, no failure.
Look for:
- Backgrounded/detached children:
... &,nohup ...,setsid. - Pipes or redirections holding stdout/stderr open (a daemonized child that inherited the log pipe keeps the phase "running").
- Interactive prompts/pagers: Django's
loaddata/migratewith a pager or input prompt can block waiting on stdin — add--no-input, setPAGER=cat, and ensure stdin is closed. - A missing explicit exit: end the script with
exit 0so the shell can't linger.
Quick test: run the exact pre-deploy command locally with stdout/stderr captured and confirm the shell (not just the SQL) terminates in seconds with exit 0. If it hangs locally, that's the bug.
2. Check for build-queue slot contention
Pre-deploy "occupies a slot in your build queue." If an earlier deployment is stuck in a pending/queued state, it can hold the slot and later deployments queue behind it — which would explain multiple consecutive deployments showing the same symptom. Cancel/remove the stuck deployments (95f23b02, 13d6da40, 6899f15b) and redeploy fresh to see if a clean slate promotes.
3. If both are clean → control-plane acknowledgment (your hypothesis), Railway must unstick
A pre-deploy that demonstrably exits 0 with no queue contention should transition to Active; if it doesn't, the completion event isn't being acknowledged upstream. You've done the right thing by including the deployment IDs — ask Railway to check the completion event for 6899f15b and promote it manually if the command genuinely completed.
4. Decouple deploy promotion from DB work (the robust fix)
Consider moving migrations out of pre-deploy into a one-shot migration service (triggered from CI, like a db-migrate service that runs migrations and exits). This is the community-standard Railway pattern: pre-deploy keeps the deploy pipeline simple, migration failures fail a job instead of wedging a deployment, and you get clean, explicit retry semantics. If you keep migrations in pre-deploy, keep them minimal and idempotent (IF NOT EXISTS, versioned migrations) — which you've already done.
TL;DR: Verify the pre-deploy shell actually exits (backgrounded children/pagers are the classic cause), then check for stuck deployments hogging the build-queue slot; if both are clean, it's the control plane not acknowledging completion — Railway needs to check the completion event and promote 6899f15b. Long-term, move migrations to a one-shot migration service so deploys never depend on pre-deploy.