2 months ago
Given a cached zstd entry, the edge serves it to every request whose Accept-Encoding lacks zstd.
Safari/iOS sends gzip, deflate, br then gets the raw zstd frame and renders full-page binary garbage.
A single curl only looks intermittent because one client samples one PoP's cache state at one moment.
Mechanism: origin (Caddy) negotiates correctly and sets Vary: Accept-Encoding. The edge buckets Accept-Encoding too coarsely —gzip,deflate,br / br / identity / empty all collapse into the same bucket as zstd, and the bucket serves whichever variant warmed it first.
Captured (asset https://www.hiphiphip.app/assets/index-K-1UCDw4.js, x-cache: HIT):
gzip, deflate, br -> content-encoding: zstd ✗ client never sent zstd
gzip, deflate -> content-encoding: gzip ✓
br -> content-encoding: zstd ✗
identity / empty -> content-encoding: zstd ✗
gzip -> content-encoding: gzip ✓
Deterministic repro (fires the instant the edge has the key cached):
URL='https://www.hiphiphip.app/assets/index-K-1UCDw4.js?cb=UNIQUE'
curl -sI -H 'Accept-Encoding: zstd' "$URL" # warm bucket → zstd, cached
curl -sI -H 'Accept-Encoding: gzip, deflate, br' "$URL" # HIT → served zstd ← bug
Note: we currently see x-cache: MISS on every request to that PoP (cdg1) — looks like edge caching is cold/off for it right now;
was returning HITs earlier today. Did edge caching/compression change recently?
Impact: ~20% of global traffic (all Safari/iOS pre-26.3) hits full-page garbage on any zstd-cached asset/document.
Workaround live: we disabled zstd at origin (gzip only)
Project-Id: 65aff0db-6586-4be0-8420-b2e67ae4378d
Deployment with zstd enabled: c651d4e2-c21d-41a8-a12d-cc44005dd46e
Attachments
11 Replies
2 months ago
Thanks for the report, I'm looking into it.
2 months ago
This should be fixed fleet-wide. We now normalize valid Accept-Encoding values before sending the request upstream, to match our cacheable encoding buckets.
2 months ago
Thank you again for the detailed report!
2 months ago
Are you confirming that this is fixed?
2 months ago
Sounds good, thanks!
Instead of waiting I tested the CDN change on a dedicated env: https://railway.com/project/65aff0db-6586-4be0-8420-b2e67ae4378d/service/8e8e4d50-be24-4b86-be1e-31b672fb1e55?environmentId=9d558642-6565-4b4f-96e7-4148e258b0c5
Here are the results I get:
Accept-Encoding contains gzip OR br → gzip (all real browsers)
gzip, deflate, br, zstd (Chrome) → gzip
gzip, deflate, br (Safari) → gzip ← gate PASS, no garbage
Accept-Encoding without gzip/br → zstd
zstd / zstd, deflate → zstd (no real browser sends this)So even browsers accepting zstd now get gzip if they accept it, making zstd support pointless.
Is it the behavior you expect?
2 months ago
Yes that's correct, we prioritize gzip in most cases which is the same behavior as other CDNs like Cloudflare
2 months ago
You may also notice that if the client only sends zstd (which would never be the case on a normal client), then we bypass the cache entirely. I have a task to better support clients that want to prioritize zstd, but the solution I implemented yesterday should fix the garbled content which caused the original issue.
Yes, there is no issue anymore on legacy Iphones. I will stick to gzip in my codebase though. Is there some place i can track your zstd support?