18 days ago
Builder VM assigned to my "worker" service has been wedged for ~20 minutes with failed to create lease: write /var/lib/buildkit/runc-overlayfs/containerdmeta.db: no space left at the [internal] load build definition step — before BuildKit reads our Dockerfile. Railway's auto-diagnose has flagged this as infrastructure. 4+ consecutive failures, retries hitting the same wedged VM. Please reassign or GC. Project: sundi133/pepper, service: worker, region: US West.
1 Replies
Status changed to Awaiting Railway Response Railway • 18 days ago
17 days ago
Hey, apologies for the disruption.
Following yesterday's outage, we worked through a heavy backlog of queued builds and deployments today. We also identified and rolled out a fix for an issue causing some builders to run out of storage causing new deployments to fail. The combination of these issues led to slow and unreliable deploys.
Both issues are now resolved: the queue has fully drained and builds and deployments are processing normally across all plans. If you have a build that's still stuck or failing, trigger a redeploy from the dashboard or CLI and it should go through.
More context on the recovery effort here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Thanks for your patience throughout this. Let us know if anything still looks off on your end.
Status changed to Awaiting User Response Railway • 17 days ago
Status changed to Solved mykal • 17 days ago