All builds failing in us-west2 (buildkitd unavailable / I/O error) - production blocked on env var change
takatamakata
PROOP

28 minutes ago

Hi Railway team,

Every build in our project has been failing since this morning, and it is blocking a production fix.

Project: zestful-creativity (38274c99-82eb-488d-9bfd-160e3a24bd6e), environment: production

Region: us-west2

Affected services: Backend (Dockerfile builder, root /clinic-analytics/backend) and cron-messages

Repo: takatamakata/EcemClinic, branch main. The commit is unchanged: it is the same commit that built and deployed successfully yesterday (current ACTIVE deployment, PR #158).

What changed: only environment variables (rotated an API credential). No code changes.

Failures (all on 2026-10-28, GMT+3):

  1. Backend, deployment c58a133e (10:36): Build > Build image ran 4m32s with no output beyond "All aboard, next stop: builder v4", then "Failed to build an image".

  2. cron-messages (around 10:37): "[4/6] COPY requirements.txt ... failed to copy files: userspace copy failed: read /requirements.txt: input/output error".

  3. Backend, deployment daf6923c (10:48, redeploy): snapshot fetched and unpacked fine, then at 10:50:08: "Build Failed: build daemon returned an error < listing workers for Build: failed to list workers: Unavailable: connection error: desc = error reading server preface: raw-read unix @->/run/buildkit/buildkitd.sock: use of closed network connection >"

  4. cron-messages redeploy failed again as well.

Your own diagnosis agent also concluded it is not caused by our code or settings.

Impact: our clinic's WhatsApp messaging (appointment confirmations and reminders) is down until the new variable values reach the running containers. The currently running deployments still hold the old, revoked credential.

Requests:

  • Please check the builders / buildkit daemon serving our builds in us-west2.
    • If builds cannot be restored quickly, is there a way to roll the currently ACTIVE (already built) image with the updated service variables, without a rebuild? That alone would fix our outage.

Thank you!

Awaiting Railway Response

0 Replies

Status changed to Awaiting Railway Response Railway • 25 minutes ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...