22 days ago
Project: CoreCRM (1e3952e4-0159-4989-a2db-5d471427242f)
Environment: staging (eccb8fd7-8ccf-4f94-a012-96aac1cff8bc), services api and worker
Since about 13:40 UTC on Oct 28, most builds for these services fail at the start of the build, before any of our build steps run:
🚂 All aboard, next stop: builder v4
[INFO] fetching snapshot
[INFO] unpacking archive
Build Failed: build daemon returned an error < listing workers for Build: failed to list workers: Unavailable: connection error: desc = "error reading server preface: raw-read unix @->/run/buildkit/buildkitd.sock: use of closed network connection" >
Failed deployments include:
api: e067c2b3-8db7-46c6-a5a1-bafe0386c8bb (13:42 UTC), then the redeploys at 13:55 and 14:04 UTC
worker: d820d3d4-336b-4aff-91fa-7529a7289543 (13:42 UTC), then the redeploy at 13:55 UTC
It’s intermittent. The worker redeploy at 14:04 UTC got a builder, but the api redeploys keep failing. We hit the same error once on Oct 27, and a single redeploy fixed it then. The same commit (52b55cb) builds cleanly with our Dockerfile in GitHub Actions, so this isn’t caused by the code. Production deploys are blocked behind staging.
2 Replies
Status changed to Awaiting Railway Response Railway • 22 days ago
22 days ago
This is on our side. We're seeing intermittent build failures in US East since around the time you first hit it, and the "failed to list workers" error is that issue, not your code or Dockerfile. The same commit will build once it clears.
We've opened an incident and you can track it here: https://status.railway.com/incident/TX3M5SWF
Running services aren't affected. Retries are hit and miss until it's resolved, so I'd wait for the incident to move to monitoring before redeploying.
Status changed to Awaiting User Response Railway • 22 days ago
22 days ago
Seeing the same on our side. Project civicgives-dev, environment production (our dev project). Same error at the image build step within 30 to 40 seconds: "error reading server preface ... buildkitd.sock: use of closed network connection". payment-reconcile failed at 13:16 and 14:26 UTC today, and clear-settled-intents at 14:26 UTC, while other services in the project built the same commit fine. One manual redeploy fixed each (13:44, 15:09 and 15:13 UTC). We also saw the same error on Oct 25, 26 and 27, so it may predate today's incident. Will follow TX3M5SWF.
Status changed to Awaiting Railway Response Railway • 22 days ago
Status changed to Awaiting User Response dizzydes90 • 22 days ago
3 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 3 days ago