a month ago
Project: insightful-magic
Project ID: 12191523-8f23-4fa0-aee4-8bf9390032f7
Environment: production
Region: sfo
Service: Postgres (service ID 1d370bec-2712-4116-9634-3c8eca06e525)
Production Postgres stuck QUEUED ("Waiting for deployment slot")
since 13:52 UTC — 15+ minutes. cms-beat also queued.
Sequence:
-
Postgres volume filled, was crash-looping with ENOSPC on pg_wal.
Volume now reports 490MB / 1000MB.
-
Every new deployment since (manual redeploy + restart) goes
straight to QUEUED and never starts.
Stuck zombie deployments (visible via GraphQL, hidden from UI):
-
staging/web from 08:49:57 UTC, status=DEPLOYING, stopped=true
-
staging/web from 08:50:11 UTC, status=DEPLOYING, stopped=true,
queuedReason="Waiting for build slot"
-
staging/web from 13:31:16 UTC, status=DEPLOYING, stopped=true
These show as removed in the UI, railway down succeeds, but they
persist in activeDeployments and appear to hold the build slot.
Public site (cms.myrugby.app) is still serving traffic on the prior
deploy, but all CMS DB-backed writes and Celery beat are down.
Please force-clear the stuck deployments so production Postgres
can come back up. This started just before the platform incidents
were declared today — may be related fallout.
1 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Thanks for the precise diagnosis — you're right that the zombie deployments (showing as removed in the UI but still present in activeDeployments) are holding the slot and blocking new deploys from leaving QUEUED. That's consistent with the platform-side fallout from today's incident.
Your data volume is intact and your public site is still serving on the prior deploy. Force-clearing those wedged deployments so the slot frees up is a platform-side action — it can't be done reliably from your end (railway down reports success but they persist). We're escalating it now with your project and service IDs (insightful-magic, Postgres 1d370bec-2712-4116-9634-3c8eca06e525). I don't have a firm ETA yet but will follow up here once the slot is cleared.
Status changed to Awaiting User Response Railway • about 1 month ago
25 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 25 days ago