Skip to content
Builds failing on builder v4: “failed to list workers … buildkitd.sock: use of closed network connection”
nullfactor10
OP

22 days ago

Project: CoreCRM (1e3952e4-0159-4989-a2db-5d471427242f)

Environment: staging (eccb8fd7-8ccf-4f94-a012-96aac1cff8bc), services api and worker

Since about 13:40 UTC on Oct 28, most builds for these services fail at the start of the build, before any of our build steps run:

🚂 All aboard, next stop: builder v4

[INFO] fetching snapshot

[INFO] unpacking archive

Build Failed: build daemon returned an error < listing workers for Build: failed to list workers: Unavailable: connection error: desc = "error reading server preface: raw-read unix @->/run/buildkit/buildkitd.sock: use of closed network connection" >

Failed deployments include:

api: e067c2b3-8db7-46c6-a5a1-bafe0386c8bb (13:42 UTC), then the redeploys at 13:55 and 14:04 UTC

worker: d820d3d4-336b-4aff-91fa-7529a7289543 (13:42 UTC), then the redeploy at 13:55 UTC

It’s intermittent. The worker redeploy at 14:04 UTC got a builder, but the api redeploys keep failing. We hit the same error once on Oct 27, and a single redeploy fixed it then. The same commit (52b55cb) builds cleanly with our Dockerfile in GitHub Actions, so this isn’t caused by the code. Production deploys are blocked behind staging.

Solved

2 Replies

Status changed to Awaiting Railway Response Railway • 22 days ago


dizzydes90
EMPLOYEE

22 days ago

This is on our side. We're seeing intermittent build failures in US East since around the time you first hit it, and the "failed to list workers" error is that issue, not your code or Dockerfile. The same commit will build once it clears.

We've opened an incident and you can track it here: https://status.railway.com/incident/TX3M5SWF

Running services aren't affected. Retries are hit and miss until it's resolved, so I'd wait for the incident to move to monitoring before redeploying.


Status changed to Awaiting User Response Railway • 22 days ago


marshallabney-ai

22 days ago

Seeing the same on our side. Project civicgives-dev, environment production (our dev project). Same error at the image build step within 30 to 40 seconds: "error reading server preface ... buildkitd.sock: use of closed network connection". payment-reconcile failed at 13:16 and 14:26 UTC today, and clear-settled-intents at 14:26 UTC, while other services in the project built the same commit fine. One manual redeploy fixed each (13:44, 15:09 and 15:13 UTC). We also saw the same error on Oct 25, 26 and 27, so it may predate today's incident. Will follow TX3M5SWF.


Status changed to Awaiting Railway Response Railway • 22 days ago


Status changed to Awaiting User Response dizzydes90 • 22 days ago


Railway
BOT

3 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 3 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...