Our web service has hit 3 consecutive build failures since 2026-04-23 22:11 UTC, all on the same Metal builder: production-builderv3-europe-west4-lk9b.

The builder is progressively degrading. Time to complete load metadata for docker.io/library/python:3.13-slim-bookworm (a Docker Hub manifest fetch that should be sub-second):

- Healthy builder, 2026-04-22: 0.1s

- lk9b attempt 1 (22:11 UTC): 7.7s

- lk9b attempt 2 (22:37 UTC): 197.7s (~26× slower)

- lk9b attempt 3 (23:22 UTC): 583.6s (~5,800× slower)

Failure signatures differ per attempt but all happen in BuildKit phases before our code enters the build context:

- Attempt 1: apt ran normally (~32s), then 39 minutes of silent layer-commit → DeadlineExceeded

- Attempts 2 & 3: stalled at metadata load, never reached apt

Dockerfile and requirements.txt byte-identical across the successful 04-22 build and all three failures. The stall is in BuildKit phases that depend only on base image + Dockerfile, not our source.

Deployment IDs:

- 59e9b551-e93d-4c6c-9f1c-ece9f5859005 (last success, 46s)

- 597c5117-7512-46f0-a8b8-6bb2d5bcb17a (fail 1, DeadlineExceeded at 40m)

- 61d930b2-338b-4fd3-b349-30a21ca78e22 (fail 2, 3m metadata stall)

- c4b3be77-19f7-4453-9e03-08d93ba01bc5 (fail 3, 9m 44s metadata stall)

Service: web (production env, project extraordinary-celebration)

Questions for the community:

1. Has anyone else seen this pattern on lk9b or other eu-west4 builders?

2. Does builder affinity rotate automatically after idle time, or are we pinned until manual intervention?

3. Any way to force a builder reassignment from our side (service redeploy, env toggle, etc.)?

Happy to share any further logs. Can't deploy until this resolves.