25 days ago
All builds for my service have been failing for ~30 min — the build node
is out of disk space. The build never reaches my app code; it dies during
Railpack's apt-get install of libatomic1.
Project: StratumFinancialPartners (stratumfp.com)
Region: US West
Builder: production-builderv3-us-west1-bjv4
Builder version: Railpack v0.23.0
Failed deploys: 2 in a row (latest PR merges to main)
Last successful deploy: ~30 min ago on the SAME builder
Build log:
install apt packages: libatomic1
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Err:1 http://deb.debian.org/debian bookworm InRelease
Error writing to file - write (28: No space left on device)...
E: Unable to locate package libatomic1
Build Failed: build daemon returned an error < failed to solve:
process "sh -c apt-get update && apt-get install -y libatomic1"
did not complete successfully: exit code: 100 >Can someone cycle this builder or route my builds to a healthy node?
Empty retry commits keep landing on the same bad host.
4 Replies
Status changed to Awaiting Railway Response Railway • 25 days ago
25 days ago
Workaround: switching region to US East fixed it — looks like the US West builder production-builderv3-us-west1-bjv4 is still broken.
erindohan
Workaround: switching region to US East fixed it — looks like the US West builder production-builderv3-us-west1-bjv4 is still broken.
25 days ago
Im going to try.
25 days ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
Status changed to Awaiting User Response Railway • 25 days ago
25 days ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
Status changed to Solved Railway • 25 days ago