a month ago
Project: civix (da1bc68b-95b2-458c-88e8-f2eb3d03c8b0)
Service: civix (dff377d3-6d69-46af-8d6b-ce517dbbc860)
Environment: production (ae0df636-0991-4fc9-930d-3ce3d4c86d84)
Every deploy attempt for the last 90 minutes has failed at the same step:
"install apt packages: libatomic1" → "Error writing to file (28: No space
left on device)" → "E: Unable to locate package libatomic1" →
"Build Failed: exit code 100".
Builder consistently scheduled to: production-builderv3-us-west1-20fh
Failed deploys: 117ffc52, plus 3 prior attempts today.
Latest commit: f2a40f8 (main branch).
This is launch day. We need either:
(a) the us-west1-20fh builder VM drained/recycled, OR
(b) our service pinned to a different builder pool.
The acknowledged "Builds are slow to progress" incident notice is up but
we are completely blocked, not just slow.
2 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 28 days ago