11 days ago
Deployment ID: e0d12c96-1225-4d3b-a22f-1af4e79149a1
Service: Production API (dodopok/estevao-api)
Project: affectionate-cat
Environment: production (us-west2 region)
ISSUE:
The build completed successfully and image was pushed at 03:17:10, but the deployment is stuck in CREATE_CONTAINER stage since 03:17:11 (over 5 minutes elapsed). Container creation typically takes only seconds, so this is abnormal.
DEPLOYMENT TIMELINE:
- 03:16:38 - Deployment triggered (PR #185 merged)
- 03:17:10 - Build completed, image pushed successfully
- 03:17:11 - Stuck in CREATE_CONTAINER (still stuck - abnormal delay)
WHAT WAS DEPLOYED:
Simple configuration change to increase Redis timeout from 1 second to 5 seconds in config/environments/production.rb - no code logic changes or dependency updates.
DIAGNOSTICS PERFORMED:
✓ No platform-wide incidents on status page
✓ No maintenance notices affecting workspace
✓ Build logs are clean (no errors)
✓ All previous deployment stages completed successfully
✓ Service resource usage is normal (Redis: 0.4% CPU, 52MB memory)
✓ Same region (us-west2) for both services
ROOT CAUSE:
This appears to be a Railway infrastructure issue - the deployment hangs after successfully building and pushing the image, specifically during the container orchestration stage.
REQUEST:
Please investigate why container creation is hanging for this deployment and either:
- Automatically retry/unstick the container, or
- Free the stuck container so we can redeploy
5 Replies
11 days ago
That deployment is currently showing as RUNNING, so the container creation stage completed and the deployment has progressed past the point where it appeared stuck. Brief delays in container creation can occur but are transient and resolve on their own, which appears to be what happened here.
Status changed to Awaiting User Response Railway • 11 days ago
11 days ago
I disagree. The first deployment (e0d12c96) was stuck in CREATE_CONTAINER for MORE THAN 5 MINUTES (from 03:17:11 to now, 8+ minutes), not a "brief delay".
This is not a normal transient delay - container creation typically takes seconds, not minutes. The fact that the second deployment (45c0b593) eventually progressed does not mean the first problem "resolved on its own" - it means the queue was congested or there was a bad state blocking that specific deployment.
THIS IS URGENT because:
- Blocks production deployments - I cannot deploy hotfixes quickly if deployments hang for 5+ minutes
- Affects system reliability - if this recurs during peak hours, it could prevent critical rollbacks
- The pattern is concerning - multiple consecutive deployments failing at the same stage signals infrastructure problems, not random transience
I request that you:
- Review infrastructure logs for the container orchestration system during the e0d12c96 deployment timeframe
- Identify what caused the 5+ minute hang
- Confirm whether there is a capacity issue or state leak in your deployment system
This is not a false alarm - it requires real investigation.
Status changed to Awaiting Railway Response Railway • 11 days ago
11 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 11 days ago
11 days ago
This confirms it IS an infrastructure problem. BOTH deployments (e0d12c96 and 45c0b593) are now stuck in CREATE_CONTAINER at the exact same stage. This is not random transience - this is a pattern indicating a systematic failure in your container orchestration layer.
First deployment: stuck 03:17:11 onwards (5+ minutes)
Second deployment: stuck 03:24:20 onwards (still pending)
Same failure point = same root cause = YOUR INFRASTRUCTURE
This requires immediate escalation to your infrastructure team. Container creation should take seconds, not minutes. Something is broken in your deployment pipeline.
URGENT: This is blocking ALL production deployments to this service and must be fixed NOW.
11 days ago
CRITICAL - INFRASTRUCTURE FAILURE CONFIRMED.
All 3 consecutive deployments are STUCK in CREATE_CONTAINER at exact same stage:
- Deploy 1 (e0d12c96): stuck 03:17:11 onwards
- Deploy 2 (45c0b593): stuck 03:24:20 onwards
- Deploy 3 (829aa97b): stuck 03:30:19 onwards (STILL PENDING)
3 of 3 = 100% failure rate on container creation = SYSTEMATIC INFRASTRUCTURE PROBLEM.
This is NOT random transience. This is YOUR DEPLOYMENT PIPELINE BROKEN.
URGENT ACTION REQUIRED:
- Immediately escalate to Infrastructure/Kubernetes team
- Check container orchestration logs for stuck requests
- Identify what's blocking CREATE_CONTAINER operations
- This is BLOCKING ALL PRODUCTION DEPLOYMENTS
We cannot work until this is fixed.
11 days ago
URGENT - PRODUCTION OUTAGE NOW
The deployment infrastructure issue has escalated to a complete service outage:
- Production API has 0 running replicas
- All user requests returning 502 "connection refused"
- System has been down for 5+ minutes
- Cannot redeploy because CREATE_CONTAINER infrastructure is broken (3 consecutive deployments stuck)
This is no longer a deployment delay - this is a PRODUCTION SERVICE DOWN situation.
I NEED immediate action from Railway infrastructure team:
- Diagnose why CREATE_CONTAINER is failing on all deployments
- Unblock container creation so I can restore service
- This requires human engineering support NOW - not automated responses
ETA for fix needed immediately.