deno cache
in Docker builds repeatedly fails with a brotli decompression error on the SAME deno
calvinrose18938-create
PROOP

2 months ago

Railway Support Ticket — persistent brotli corruption fetching deno.land/std from the build environment

Persistent brotli corruption fetching deno.land/std in Docker builds — 6 failures today, same file every time

Environment: staging. Services ft-advance (cron, worst hit) and

proud-vision, both building the same railway-worker/Dockerfile

(root = repo root, denoland/deno:2.8.1, RUN deno cache … fetches ~290

remote modules).

The error — byte-identical in all 6 failed builds

error: Import 'https://deno.land/std@0.132.0/node/internal/streams/from.mjs' failed.
    0: brotli error
    at https://deno.land/std@0.132.0/node/internal/streams/duplex.mjs:23:19
[ERRO] [9/9] RUN deno cache main.ts forwardTestJob.ts dispatcher.ts
Build Failed: ... exit code: 1

Always this one file; the ~290 sibling downloads in the same builds succeed.

Full logs: railway logs --build <deployment-id> (IDs below).

Build matrix (2026-08-07, UTC)

| Service | Deployment | Commit | Time | deno fetch | Result |

|---|---|---|---|---|---|

| ft-advance | d0b5db71 | 8190340 | 13:14 | fresh | FAIL (brotli) |

| ft-advance | 5a2cd2ec | 8190340 | 13:33 | fresh (manual retry) | OK |

| proud-vision | 68cf9400 | d328852 | 13:35 | fresh | OK |

| ft-advance | 679b7dc8 | d328852 | 13:35 | fresh | FAIL (brotli) |

| proud-vision | 28574da9 | eaefbd1 | 14:13 | fresh | OK |

| ft-advance | d68e4458 | eaefbd1 | 14:13 | fresh | FAIL (brotli) |

| proud-vision | 2c8ee4e7 | 4687229 | 14:25 | fresh (Dockerfile edit broke cache) | FAIL (brotli) |

| ft-advance | 18181bbb | 4687229 | 14:25 | fresh | FAIL (brotli) |

| proud-vision | 8cc887c8 | ef441c2 | 14:27 | layer-cached, 0 downloads | OK |

| ft-advance | 9567cf5c | ef441c2 | 14:27 | fresh | FAIL (brotli) |

Reading: identical Dockerfile + commits on both services, so not commit

content. ft-advance fails 5/6 fresh fetches; proud-vision 1/3 — and its

final "green" downloaded nothing (layer cache masked it). Correlation is

with which builder/egress path each service's build lands on. Fetching the

URL from outside Railway is clean (curl with Accept-Encoding: br: HTTP

200, decodes to the correct 918-byte module) — a cache/edge on your egress

path is serving a corrupted brotli object for this one file.

No retry path

The dashboard retry control (used for the 13:33 success) is no longer

offered on the failed deployments, and `railway redeploy --service

ft-advance --yes` refuses: "The latest deployment … cannot be redeployed …

currently building, deploying, or was removed" (latest = FAILED 9567cf5c;

nothing was building).

Asks

  1. Flush/invalidate the build-env cache or egress path serving the corrupt

    object for https://deno.land/std@0.132.0/node/internal/streams/from.mjs,

    or reassign the affected builder (ft-advance seems pinned to the bad one).

  2. Restore a retry/redeploy control for failed deployments (dashboard and

    CLI both refuse, above).

  3. Confirm if this is a known deno.land CDN incident on your side.

Impact: the cron service is stuck on the 13:33 build; we cannot ship newer

commits to it.

$20 Bounty

3 Replies

Railway
BOT

2 months ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 2 months ago


calvinrose18938-create
PROOP

2 months ago

Thank you for the quick response! The ticket has been updated upon new evidence. Please review it and respond.


calvinrose18938-create
PROOP

2 months ago

Thank you for the helpful and quick responses! Neither quite fits the evidence though: every failed build ran in a fresh container with an empty DENO_DIR, so there was no local cache to bust — Deno fetched the URL fresh each time and got corrupt bytes. And we'd rather not relocate the whole environment. We've since added a retry loop around deno cache, which both mitigates and proves the point: today at 16:30Z, attempt 1 on the same URL failed with the brotli error and attempt 2 seconds later fetched it clean — intermittent corruption on the builder egress path for that one object. Still hoping someone from Railway infra can take a look at it. Full logs from all five failed builds available.


rluf
FREE

a month ago

Hi — you've already proven the important part (fresh container, empty DENO_DIR, same URL fails then succeeds seconds later → it's the fetch path, not your code). While the builder-egress side is the infra team's to look at, there is a change on your side that removes the problem entirely, because it removes the fetch from the build: stop downloading deno.land/std at build time at all.

  1. Vendor the remote modules and commit them. Deno supports this natively — in deno.json:

json

{ "vendor": true }

Then locally run deno install --entrypoint main.ts (deno cache is deprecated in Deno 2; deno install --entrypoint is its replacement). Deno writes every remote module into a vendor/ directory next to your code. Commit that directory. From then on, builds resolve https://deno.land/std@0.132.0/... from vendor/ and never touch the network for those imports — no brotli, no CDN, no egress path. The Deno docs describe "vendor": true specifically as the option for "CI servers, Docker containers" for exactly this reason.

  1. Lock it so drift fails loudly instead of silently re-fetching. Commit deno.lock and build with the frozen flag, so a new or changed dependency fails the build rather than triggering a fresh download mid-build:

dockerfile

FROM denoland/deno:2.8.1

WORKDIR /app

COPY deno.json deno.lock ./

COPY vendor ./vendor

COPY . .

RUN deno install --frozen --entrypoint main.ts

CMD ["deno", "run", "-A", "main.ts"]

(Deno 2.8 also has deno ci --prod for the install step; either works.)

  1. The failing import itself is worth a look. std@0.132.0/node/internal/streams/from.mjs is the 2022-era Node polyfill layer of the old std library. In Deno 2 the node: built-ins are native, and std moved to JSR (jsr:@std/..., served from jsr.io, a different CDN than deno.land). Something in your dependency graph is still pulling that very old path — deno info main.ts | grep 0.132 will show which import brings it in. Upgrading that one dependency (or replacing https://deno.land/std@0.132.0/node/... with the node: equivalent) drops ~290 deno.land fetches from your build and makes step 1 smaller too.

Net effect: the build becomes deterministic and offline with respect to deno.land, the retry loop can go, and if the builder egress really does have an intermittent brotli issue, it simply stops being your problem. Happy to help trace the 0.132.0 import if deno info output is unclear.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...