13 days ago
Given a cached zstd entry, the edge serves it to every request whose Accept-Encoding lacks zstd.
Safari/iOS sends gzip, deflate, br then gets the raw zstd frame and renders full-page binary garbage.
A single curl only looks intermittent because one client samples one PoP's cache state at one moment.
Mechanism: origin (Caddy) negotiates correctly and sets Vary: Accept-Encoding. The edge buckets Accept-Encoding too coarsely —gzip,deflate,br / br / identity / empty all collapse into the same bucket as zstd, and the bucket serves whichever variant warmed it first.
Captured (asset https://www.hiphiphip.app/assets/index-K-1UCDw4.js, x-cache: HIT):
gzip, deflate, br -> content-encoding: zstd ✗ client never sent zstd
gzip, deflate -> content-encoding: gzip ✓
br -> content-encoding: zstd ✗
identity / empty -> content-encoding: zstd ✗
gzip -> content-encoding: gzip ✓
Deterministic repro (fires the instant the edge has the key cached):
URL='https://www.hiphiphip.app/assets/index-K-1UCDw4.js?cb=UNIQUE'
curl -sI -H 'Accept-Encoding: zstd' "$URL" # warm bucket → zstd, cached
curl -sI -H 'Accept-Encoding: gzip, deflate, br' "$URL" # HIT → served zstd ← bug
Note: we currently see x-cache: MISS on every request to that PoP (cdg1) — looks like edge caching is cold/off for it right now;
was returning HITs earlier today. Did edge caching/compression change recently?
Impact: ~20% of global traffic (all Safari/iOS pre-26.3) hits full-page garbage on any zstd-cached asset/document.
Workaround live: we disabled zstd at origin (gzip only)
Project-Id: 65aff0db-6586-4be0-8420-b2e67ae4378d
Deployment with zstd enabled: c651d4e2-c21d-41a8-a12d-cc44005dd46e
Attachments
11 Replies
13 days ago
Thanks for the report, I'm looking into it.
13 days ago
This should be fixed fleet-wide. We now normalize valid Accept-Encoding values before sending the request upstream, to match our cacheable encoding buckets.
13 days ago
Thank you again for the detailed report!
12 days ago
Are you confirming that this is fixed?
12 days ago
Sounds good, thanks!
Instead of waiting I tested the CDN change on a dedicated env: https://railway.com/project/65aff0db-6586-4be0-8420-b2e67ae4378d/service/8e8e4d50-be24-4b86-be1e-31b672fb1e55?environmentId=9d558642-6565-4b4f-96e7-4148e258b0c5
Here are the results I get:
Accept-Encoding contains gzip OR br → gzip (all real browsers)
gzip, deflate, br, zstd (Chrome) → gzip
gzip, deflate, br (Safari) → gzip ← gate PASS, no garbage
Accept-Encoding without gzip/br → zstd
zstd / zstd, deflate → zstd (no real browser sends this)So even browsers accepting zstd now get gzip if they accept it, making zstd support pointless.
Is it the behavior you expect?
12 days ago
Yes that's correct, we prioritize gzip in most cases which is the same behavior as other CDNs like Cloudflare
12 days ago
You may also notice that if the client only sends zstd (which would never be the case on a normal client), then we bypass the cache entirely. I have a task to better support clients that want to prioritize zstd, but the solution I implemented yesterday should fix the garbled content which caused the original issue.
Yes, there is no issue anymore on legacy Iphones. I will stick to gzip in my codebase though. Is there some place i can track your zstd support?