13 days ago
Hello,
Since 2026-09-23 about 10:30 UTC, our production builds take 5–8 min. They took about 2 min before. The build logs show that each build now runs on a different production-builderv3-europe-west4-* host, and no layers come from the cache. We made no change to our build configuration in this period.
Evidence (web service)
The build time is the time from the first build log line to the last.
+--------------------------------------+-------------+----------------------------------------+-------+------------+
| Deployment ID | Start (UTC) | Builder host | Cache | Build time |
+--------------------------------------+-------------+----------------------------------------+-------+------------+
| 5b8b6efd-5dd5-4cbe-8b20-14edc28560e8 | 09-20 13:03 | builder-xiwwla | hit | 1m54s |
| c53a4fb5-4b42-4471-bce1-45119696e188 | 09-21 15:47 | builder-xiwwla | hit | 4m43s |
| dac92171-79ff-4b3f-a210-7a0c98dd59bb | 09-22 10:43 | builder-oohdpi | MISS | 8m38s |
| 1c37048f-0f43-4f6f-97ad-26fc149d9c34 | 09-22 15:51 | builder-xiwwla | hit | 2m03s |
| a2d54914-d40f-48d6-b900-788670dc4589 | 09-22 20:48 | builder-xiwwla | hit | 2m10s |
| 3078a44e-5bad-4cc1-9636-6db31f4d13da | 09-23 10:30 | production-builderv3-europe-west4-pw5v | MISS | 4m51s |
| 6640238e-f91c-4500-9a01-7a6efc531789 | 09-23 10:41 | production-builderv3-europe-west4-p3p3 | MISS | 7m46s |
| 4b89a341-009f-410b-a36b-f45b723de1f6 | 09-23 11:23 | production-builderv3-europe-west4-ccf9 | MISS | 8m07s |
+--------------------------------------+-------------+----------------------------------------+-------+------------+
A build got a cache hit only when it ran on the same host as the previous build.
Why this is a cache loss and not a change in our code
On every miss, the step install mise packages: node, ruby downloads Node 20.19.1 and Ruby 4.0.1 again. This step depends only on the base image and the tool versions, and neither of them changed. A change to our code cannot invalidate this layer.
On every miss, bundle install installs all 259 gems, although Gemfile.lock did not change between builds.
The image export is also slower on a cold host. In 4b89a341, "exporting to docker image format" took about 4 min (11:27:26 → 11:31:21). In a2d54914, which hit the cache, it took about 1 min (20:49:23 → 20:50:19).
Worker service
The worker builds show the same cold-host behaviour: apt packages, gems, and Node/Ruby all install from nothing. Two of those builds ran for 12–15 min and ended as FAILED after image push:
7109b6be-9ada-4e60-8920-87d612227c7b, builder builder-qpcbvd, 10:30:33 → 10:45:59
cd876dd1-93d6-4c90-a5ff-42278b0ce2b9, builder builder-fumiuy, 10:41:14 → 10:53:38
Questions
Does the production-builderv3 pool keep a build cache for each project? If it does, why does each of our builds go to a new host with no cache?
Is this a temporary effect of a builder migration, or the new normal behaviour?
Can you pin our project to one builder, or share the layer cache between builders?
Why did the two worker builds fail after image push? Is this a timeout on cold builds?
Thank you.
1 Replies
Status changed to Awaiting Railway Response Railway • 13 days ago
13 days ago
Our build system scales up and down with demand, so which builder a build lands on changes and a cache hit is not guaranteed, and for builds that depend on cache our docs recommend building your own image in a pipeline and deploying that image directly. The two failed worker builds did not fail after image push: both failed at the code snapshot step, which runs before the image build.
Status changed to Awaiting User Response Railway • 13 days ago
Status changed to Solved tadast • 13 days ago