a month ago
Hi team,
Following up on the GCP outage incident. All my builds since 20:13 UTC are
scheduled on builder production-builderv3-us-west1-s2tn and consistently fail:
First with Debian GPG signature errors on apt-get update
Then failed to create prepare snapshot dir: ... no space left on device
Now failed to fetch snapshot (repeated)
I have 5 deployments stuck in BUILDING state with no progress (40+ minutes).
Manual redeploys keep getting scheduled on the same broken builder.
Could you please either:
Reclaim disk / garbage-collect on production-builderv3-us-west1-s2tn, OR
Route my service to a healthy us-west1 builder
Project ID: bc6849fb-ea5d-4054-9037-3ab1443c9521
Service ID: 66dd5163-b43b-409f-adf1-c7513c5621dc
Environment ID: f981bb7a-c3a4-4554-be2c-88ee77af8522
Stuck deployment IDs:
36841001-e32f-4853-b9e9-46ea7bb41c45
5fb032e0-c833-4949-a1d9-c319f527ca04
50cc0939-b4e2-4f58-b07d-1e6fb4e9ebce
087a9438-e510-4597-b794-3d0796a47c41
32025830-1237-4110-be7f-8056011c8a7a
This is blocking a critical production fix (negative pallet count in monitoring
dashboard). Thanks!
Attachments
1 Replies
a month ago
We can confirm your latest deployment is stuck in BUILDING status and your existing deployment continues to serve traffic in the meantime. The build failures you're seeing (disk exhaustion, snapshot errors) are part of an ongoing platform-wide builder infrastructure issue that our engineering team is actively remediating, including reclaiming disk and rescheduling builds off affected nodes. We recommend cancelling the five queued builds from the service's deployments page and attempting a fresh redeploy once the builder remediation completes - stacking retries on the same node will not help while the fix is in progress.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved Railway • 27 days ago