Your build completed in about 70 seconds, but the current deployment on axiombrain has been stuck on its pre-deploy command for over 14 minutes (and counting) without exiting. A pre-deploy command that never exits holds its deploy slot indefinitely, which is what makes the deploy appear to take over an hour. Cancel the stuck deployment, then check what the pre-deploy command is doing - the most common cause is a migration blocked on a database lock or a script that hits an error and hangs instead of exiting non-zero.

Re: axiombrain pre-deploy stuck — evidence the command exits; the exit event is not being observed

Thanks for the analysis, but I've since gathered evidence that the pre-deploy command is not what's hanging. I believe the fault is in the pre-deploy → start handoff on your side, and I can show where.

Environment: project 9266107b-038d-4a7c-a141-b2af5df58dfb (axiombrain), environment 74318eb4-0b0f-42ca-b9b0-50c9b29fb703 (production), service 5e652423-8c9a-486f-9aa3-5bfa6f779683, runtime V2.

Affected deployments: 9f3af31c-bbae-4386-be50-1099f6b385a2 (Aug 27 19:50 UTC) and f38d4dd6-9d79-4655-89c5-c357e66516d9 (Aug 27 21:31 UTC) — both wedged identically. Last good: 0d98c2ba-dab1-47ca-bfd7-df72390b985a (Aug 26 17:07 UTC).

The command completes its work. In f38d4dd6, the pre-deploy logged at 21:32:47 UTC — 2 seconds after container start:

INFO [alembic.env] alembic migration run complete; revision is now 0046_deal_classification

This run was a schema no-op (the DB was already at head). I verified the database directly: revision at head, all expected columns/constraints present, zero locks held, zero blocked sessions, zero idle-in-transaction. So neither of your suggested causes applies: it was not blocked on a database lock, and it did not error.

The command exits when run anywhere else. I built the exact deployed commit into the exact Dockerfile image locally and ran the exact pre-deploy command (python -m alembic -c mcp_server/db/alembic.ini upgrade head) against Postgres 17: exit code 0 in ~4s, four consecutive runs (both a full migration chain and the no-op case matching production). The error path also terminates promptly (exit 1 in 4s on unreachable DB).

What's missing is your exit event. Compare deploy logs:

0d98c2ba (success, Aug 26): last migration line at 17:08:26.527 → Stopping Container at 17:08:26.988 → new Starting Container → app up.

Both wedged deploys: completion line, then no Stopping Container ever — 17+ minutes of total silence, with the API reporting status: BUILDING alongside deploymentStopped: true simultaneously.

Nothing changed on my side between the good and bad deploys. The deploy manifests are byte-identical (same preDeployCommand, healthcheck, restart policy, runtime V2). The installed dependency sets differ by exactly one package (cyclopts 4.23.2→4.23.3, a patch bump not imported by the pre-deploy process). This worked ~12 consecutive times Aug 20–26 with the same configuration.

Could you check whether the pre-deploy container's exit is being observed by the orchestrator for this service — e.g., the exit-event pipeline for V2 runtime pre-deploy containers — sometime after Aug 26 17:08 UTC? Happy to trigger a fresh deploy while you watch.