a month ago
Project ID: 78216edf-19b7-42ff-b2d4-929aa5788850
Environment: production (environmentId: 1b508483-1063-483e-9b80-7c5e0ef19aaf)
Source repo is private and already connected/authorized to this Railway project, so you should have access to it via the deploy integration — happy to share the repo name/link over a private channel if that's not sufficient.
We're seeing what looks like a corrupted BuildKit cache affecting several services in this project, in three different failure modes, all starting roughly 6+ hours ago and persisting across many new commits/cache-busts since:
Two services ("giving-victory" and "fearless-unity") fail to build entirely, every single attempt, with: failed to calculate checksum of ref ::: "/{file}": not found The specific COPY line that fails changes between attempts, but the cache reference hash prefix (wjn478le97m8r5apco32sjv7d) stays the same across many rebuilds. We've tried: pushing fresh commits with Dockerfile comment changes to bust cache, and this has not helped at all — same error, same cache hash.
One service ("dependable-emotion") reports "Deployment successful" and shows the latest commit as Active, but the running container is executing OLD code. We proved this by adding a unique print() statement as the literal first line of the entrypoint script, pushing, confirming Railway marked the new commit Active, running the cron job, and the print never appeared in the logs — across three separate commits/deploys in a row. The deployed image appears to be silently reusing a stale layer despite reporting a successful new build.
One service ("upbeat-nourishment") reports "Deployment successful" for a commit that adds a new file reference via COPY in the Dockerfile, but at runtime we get: python3: can't open file '/app/crawl_houseofsuperstep.py': [Errno 2] No such file or directory The Dockerfile correctly includes COPY crawl_houseofsuperstep.py . (verified in the repo), and the build logs show "Deployment successful", but the file is missing from the built image. The build log's COPY step list appears to skip several COPY lines from the Dockerfile silently (no error shown).
All three symptoms point to the same underlying cause: build layers/cache references are corrupted or being served inconsistently, sometimes causing hard build failures (#1), sometimes silently serving stale application code (#2), and sometimes silently omitting files from the final image (#3) — all while Railway reports the build/deploy as successful.
We've already tried: cache-busting via Dockerfile comment changes, waiting several hours, and retrying multiple times. None of this has resolved it. Could you please check the BuildKit cache/builder state for this project and clear/rebuild it from scratch on your end?
Happy to provide specific deployment IDs or additional logs if useful.
4 Replies
Status changed to Awaiting Railway Response Railway • 26 days ago
a month ago
Thanks for the suggestion. I added NO_CACHE=1 to all 4 affected services and redeployed each. Results:
- giving-victory / fearless-unity: still fail immediately with the exact same checksum error (failed to calculate checksum of ref wjn478le97m8r5apco32sjv7d::...: not found). NO_CACHE=1 had no effect at all here - build logs still show every COPY step as "cached".
- dependable-emotion / upbeat-nourishment: with NO_CACHE=1, the build logs now show a genuinely fresh build (no "cached" tags anywhere, realistic per-step timings, pip install actually took ~10s instead of 0ms). So NO_CACHE=1 does force a real rebuild for these two.
- However, even with that fresh rebuild, upbeat-nourishment still fails at runtime with the exact same error as before: python3: can't open file '/app/crawl_houseofsuperstep.py': [Errno 2] No such file or directory. So the file is still missing from the final image even when the build itself is provably not using cached layers.
This suggests the corruption isn't purely a BuildKit layer-cache issue - something is wrong further down the pipeline (image export/push, or registry storage) that NO_CACHE doesn't touch. Could someone take a direct look at the upbeat-nourishment service's most recent deployment and giving-victory/fearless-unity's builder state specifically? Happy to share deployment IDs.
a month ago
Update: to rule out per-service corruption, I deleted both "giving-victory" and "fearless-unity" entirely and recreated them from scratch as brand-new services (new names: "compassionate-respect" and "illustrious-stillness"), same repo/root directory/branch/start command/cron schedule/DATABASE_URL var, connected fresh to the same GitHub repo.
Both brand-new services failed on their very first build attempt ever, with the same failure mode (failed to calculate checksum of ref ...: not found, on the schema_postgres.sql COPY step) - but with a different cache ref hash than the one in my first report (this time: wdrxbxg29avkj3e6a1d8wzlml::ua6y8r833nep45o142la). Since these are brand-new services that never had any prior cache of their own, this rules out a per-service cache namespace issue - whatever's corrupted is either a shared/content-addressed cache layer at the account or infra level, or the corruption is being freshly (re)introduced on every build rather than being one fixed stale blob. Either way, recreating services is not a viable workaround.
Given this, it would help to know: is there a way to force a full cache purge for this repo/commit on Railway's end, and any update on timeline for a fix? Happy to provide the new deployment IDs for compassionate-respect/illustrious-stillness if useful.
a month ago
Correction / closing this out: this turned out to NOT be a Railway platform bug. On our end, our backend/ directory (the Docker build context for these services) was missing 16 source files that our Dockerfile's COPY instructions expected (main.py, db_common.py, migrate.py, schema_postgres.sql, seed_data.json, and several scraper modules) - leftover from an incomplete repo restructuring on our side. BuildKit's "failed to calculate checksum of ref ...: not found" error was actually correctly reporting that the source file genuinely didn't exist in the build context, not a cache corruption issue. Recreating the services from scratch didn't fix it because the missing files were the real root cause, not any per-service cache state. We added the missing files to the repo and all previously-failing services rebuilt and deployed successfully. Apologies for the earlier reports pointing at Railway/BuildKit - appreciate the platform stability once the real bug was ours to fix. Closing this ticket.
Status changed to Solved Railway • 26 days ago