a month ago
Workspace: Ata's Projects (Pro). Since ~17:20 UTC Aug 29, every new deployment in EVERY project of this workspace builds successfully (image pushed) and then sits QUEUED with 'Waiting for deployment slot' - now 3+ hours. Verified across two unrelated projects (keppel-lp-service-dev and jason-health-backend), replica limits from 3GB to 8GB. Existing deployments keep running normally. Already ruled out: compute usage $38.99 vs a $200 hard limit; replica limits reduced from 24GB plan max to 8GB and below; stale queued deployments removed; fresh deployments re-pushed - same result, build SUCCESS then QUEUED with queuedReason 'Waiting for deployment slot' (confirmed via GraphQL API). Possibly related: ~42 'Postgres auto-updated to postgres-ssl:18' notifications rolled through this workspace over the same hours. Impact: the worker service in keppel-lp-service-dev (env dev) is OFFLINE waiting for its built deployment to place, pausing scheduled jobs. Example: worker deployment 18cccf88-9b37-4564-8cd9-632c5a6f6f8b QUEUED since 18:57 UTC.
3 Replies
a month ago
Your workspace's 10 concurrent deployment slots were saturated, most likely by the wave of 42 Postgres auto-updates that rolled through during the same window. Build/deploy concurrency is per workspace, so once all 10 slots were occupied, every other deployment across every project queued behind them. The backlog has now cleared: the deployments you cancelled freed the slots, and fresh deploys (including both the backend and worker services) are completing successfully with normal step timings as of ~19:12 UTC today.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
Your workspace's 10 concurrent deployment slots were saturated, most likely by the wave of 42 Postgres auto-updates that rolled through during the same window. Build/deploy concurrency is per workspace, so once all 10 slots were occupied, every other deployment across every project queued behind them. The backlog has now cleared: the deployments you cancelled freed the slots, and fresh deploys (including both the backend and worker services) are completing successfully with normal step timings as of ~19:12 UTC today.
a month ago
The saturation has recurred and this time cancelling does not clear it. Since ~08:47 UTC today every deployment queues again ('Waiting for deployment slot'). We have since: removed ALL queued deployments in keppel-lp-service-dev (backend/worker/worker-llm/migration-ledger-watch) and in jason-health-backend (worker/api) via deploymentRemove; reduced replica limits (backend 5GB/worker 4GB/worker-llm 5GB, jason api+worker 2GB); freed ~7GB of reserved memory; then pushed a SINGLE fresh backend deploy (project dfa192d3-3b3c-41a6-8a01-ef92ddcb7b4d, env dev). It went INITIALIZING then BUILDING then back to QUEUED and has sat there 10+ minutes as the only queued deployment we can see. Compute usage $45.88 of $200. Can you check server-side what is occupying the 10 workspace slots and clear it? This has blocked production fixes for 6+ hours today.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
To correct my earlier message, the slots are not free. All 10 of your workspace's deployment slots are held right now by 14 deployments of the migration-ledger-watch service, each stuck on its pre-deploy command since as early as 05:18 UTC today. These deployments show as BUILDING/DEPLOYING, not QUEUED, which is why removing queued deployments did not clear them. The pre-deploy command runs, outputs a migration manifest with "ok": false and 14 "unexplained_unrecorded" migrations, then never exits, so it holds its slot indefinitely. Cancel these specific migration-ledger-watch deployments via deploymentRemove (target the ones in BUILDING/DEPLOYING status, not QUEUED), and the backend deploy will start immediately. You will also want to fix the pre-deploy command so it exits non-zero on failure rather than hanging.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 30 days ago