deno cachein Docker builds repeatedly fails with a brotli decompression error on the SAME deno
2 months ago
Railway Support Ticket — persistent brotli corruption fetching deno.land/std from the build environment
Persistent brotli corruption fetching deno.land/std in Docker builds — 6 failures today, same file every time
Environment: staging. Services ft-advance (cron, worst hit) and
proud-vision, both building the same railway-worker/Dockerfile
(root = repo root, denoland/deno:2.8.1, RUN deno cache … fetches ~290
remote modules).
The error — byte-identical in all 6 failed builds
error: Import 'https://deno.land/std@0.132.0/node/internal/streams/from.mjs' failed.
0: brotli error
at https://deno.land/std@0.132.0/node/internal/streams/duplex.mjs:23:19
[ERRO] [9/9] RUN deno cache main.ts forwardTestJob.ts dispatcher.ts
Build Failed: ... exit code: 1Always this one file; the ~290 sibling downloads in the same builds succeed.
Full logs: railway logs --build <deployment-id> (IDs below).
Build matrix (2026-08-07, UTC)
| Service | Deployment | Commit | Time | deno fetch | Result |
|---|---|---|---|---|---|
| ft-advance | d0b5db71 | 8190340 | 13:14 | fresh | FAIL (brotli) |
| ft-advance | 5a2cd2ec | 8190340 | 13:33 | fresh (manual retry) | OK |
| proud-vision | 68cf9400 | d328852 | 13:35 | fresh | OK |
| ft-advance | 679b7dc8 | d328852 | 13:35 | fresh | FAIL (brotli) |
| proud-vision | 28574da9 | eaefbd1 | 14:13 | fresh | OK |
| ft-advance | d68e4458 | eaefbd1 | 14:13 | fresh | FAIL (brotli) |
| proud-vision | 2c8ee4e7 | 4687229 | 14:25 | fresh (Dockerfile edit broke cache) | FAIL (brotli) |
| ft-advance | 18181bbb | 4687229 | 14:25 | fresh | FAIL (brotli) |
| proud-vision | 8cc887c8 | ef441c2 | 14:27 | layer-cached, 0 downloads | OK |
| ft-advance | 9567cf5c | ef441c2 | 14:27 | fresh | FAIL (brotli) |
Reading: identical Dockerfile + commits on both services, so not commit
content. ft-advance fails 5/6 fresh fetches; proud-vision 1/3 — and its
final "green" downloaded nothing (layer cache masked it). Correlation is
with which builder/egress path each service's build lands on. Fetching the
URL from outside Railway is clean (curl with Accept-Encoding: br: HTTP
200, decodes to the correct 918-byte module) — a cache/edge on your egress
path is serving a corrupted brotli object for this one file.
No retry path
The dashboard retry control (used for the 13:33 success) is no longer
offered on the failed deployments, and `railway redeploy --service
ft-advance --yes` refuses: "The latest deployment … cannot be redeployed …
currently building, deploying, or was removed" (latest = FAILED 9567cf5c;
nothing was building).
Asks
-
Flush/invalidate the build-env cache or egress path serving the corrupt
object for
https://deno.land/std@0.132.0/node/internal/streams/from.mjs,or reassign the affected builder (ft-advance seems pinned to the bad one).
-
Restore a retry/redeploy control for failed deployments (dashboard and
CLI both refuse, above).
-
Confirm if this is a known deno.land CDN incident on your side.
Impact: the cron service is stuck on the 13:33 build; we cannot ship newer
commits to it.
3 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
Thank you for the quick response! The ticket has been updated upon new evidence. Please review it and respond.
2 months ago
Thank you for the helpful and quick responses! Neither quite fits the evidence though: every failed build ran in a fresh container with an empty DENO_DIR, so there was no local cache to bust — Deno fetched the URL fresh each time and got corrupt bytes. And we'd rather not relocate the whole environment. We've since added a retry loop around deno cache, which both mitigates and proves the point: today at 16:30Z, attempt 1 on the same URL failed with the brotli error and attempt 2 seconds later fetched it clean — intermittent corruption on the builder egress path for that one object. Still hoping someone from Railway infra can take a look at it. Full logs from all five failed builds available.
a month ago
Hi — you've already proven the important part (fresh container, empty DENO_DIR, same URL fails then succeeds seconds later → it's the fetch path, not your code). While the builder-egress side is the infra team's to look at, there is a change on your side that removes the problem entirely, because it removes the fetch from the build: stop downloading deno.land/std at build time at all.
- Vendor the remote modules and commit them. Deno supports this natively — in deno.json:
json
{ "vendor": true }
Then locally run deno install --entrypoint main.ts (deno cache is deprecated in Deno 2; deno install --entrypoint is its replacement). Deno writes every remote module into a vendor/ directory next to your code. Commit that directory. From then on, builds resolve https://deno.land/std@0.132.0/... from vendor/ and never touch the network for those imports — no brotli, no CDN, no egress path. The Deno docs describe "vendor": true specifically as the option for "CI servers, Docker containers" for exactly this reason.
- Lock it so drift fails loudly instead of silently re-fetching. Commit deno.lock and build with the frozen flag, so a new or changed dependency fails the build rather than triggering a fresh download mid-build:
dockerfile
FROM denoland/deno:2.8.1
WORKDIR /app
COPY deno.json deno.lock ./
COPY vendor ./vendor
COPY . .
RUN deno install --frozen --entrypoint main.ts
CMD ["deno", "run", "-A", "main.ts"]
(Deno 2.8 also has deno ci --prod for the install step; either works.)
- The failing import itself is worth a look. std@0.132.0/node/internal/streams/from.mjs is the 2022-era Node polyfill layer of the old std library. In Deno 2 the node: built-ins are native, and std moved to JSR (jsr:@std/..., served from jsr.io, a different CDN than deno.land). Something in your dependency graph is still pulling that very old path — deno info main.ts | grep 0.132 will show which import brings it in. Upgrading that one dependency (or replacing https://deno.land/std@0.132.0/node/... with the node: equivalent) drops ~290 deno.land fetches from your build and makes step 1 smaller too.
Net effect: the build becomes deterministic and offline with respect to deno.land, the retry loop can go, and if the builder egress really does have an intermittent brotli issue, it simply stops being your problem. Happy to help trace the 0.132.0 import if deno info output is unclear.