17 days ago
Between 2026-06-18 and 2026-06-19, three independent services in my account, all
in US West (California) Metal, began failing on outbound HTTPS calls. Every
failure is a truncated gzip-compressed response body, which Node surfaces as
"Premature close" / ERR_STREAM_PREMATURE_CLOSE. The exact same requests succeed
from off-Railway with identical credentials, and the upstream provider's status
page was green throughout. This points to the Railway egress path corrupting
large compressed response bodies, not the apps or the upstream.
Strong corroborating detail: the failures correlate with RESPONSE SIZE, not host.
On the same egress, small-JSON responses kept working normally (e.g. Schwab,
Alpha Vantage, Pushover calls never failed), while large gzipped response bodies
truncated. The confirmed-failing endpoint in all three apps is the one that
returns the largest compressed payloads.
The common fix across all three was to request UNCOMPRESSED responses
(Accept-Encoding: identity). Two of the three could only make that header take
effect via a custom fetch; the SDK's defaultHeaders did not reach the wire.
Affected services (all env=production, region US West / California, Metal):
-
MySupport - confirmed failing host api.anthropic.com; @anthropic-ai/sdk 0.39.0;
STREAMING (messages.stream). Onset 2026-06-19 ~11:41 UTC.
-
IdeaHub - confirmed failing host api.anthropic.com; @anthropic-ai/sdk 0.32.1;
STREAMING. Onset between last success 2026-06-19 07:33 UTC and first failure
14:31 UTC.
-
MyGuru - confirmed failing host api.anthropic.com; @anthropic-ai/sdk 0.32.1;
BUFFERED (messages.create). Earliest evidenced failure 2026-06-18 ~16:15 UTC
(logs before that not retained; true onset may be earlier).
Common error signature (Node):
"Invalid response body while trying to fetch
https://api.anthropic.com/v1/messages: Premature close"
code: ERR_STREAM_PREMATURE_CLOSE
Thrown in gzip decompression (truncated compressed body) for buffered calls,
and at the response socket for streamed calls. Affects buffered AND streaming.What we ruled out:
-
Not the upstream API: identical request + key returns HTTP 200 from a
non-Railway machine; provider status page green.
-
Not credentials: same keys work off-Railway.
-
Not app code: three independent codebases, two different SDK versions, one
buffered and two streaming, all fail identically.
-
Not container state: service restart and fresh redeploy still failed 100%.
-
Not host-specific: small responses on the same egress to other hosts were fine;
only large gzipped bodies truncated.
Workaround that resolved all three:
Force "Accept-Encoding: identity" on the outbound requests (via a custom fetch,
since defaultHeaders did not reach the wire). Uncompressed responses arrive
intact. Confirmed failing host across all three: api.anthropic.com (large
gzipped responses). Other outbound hosts were hardened with the same header as a
precaution; no failures were observed on them.Questions:
-
Any change to outbound networking / egress / NAT / routing, or any known
incident, in US West (California) Metal between 2026-06-18 and 2026-06-19?
-
Could this be a path-MTU / fragmentation issue on the egress path? The size
correlation (small responses OK, large compressed bodies truncated) is classic
PMTU blackholing.
-
Could our shared egress IPs be getting connections reset by an upstream CDN
(e.g. Cloudflare, which fronts api.anthropic.com)? If so, can we get a
dedicated/static outbound IP?
Environment: all three services build on Metal US-West (us-west1) builders, env
production. Service IDs and full stack traces available on request.
1 Replies
Status changed to Awaiting Railway Response Railway • 17 days ago
15 days ago
Thanks for the unusually thorough writeup, the isolation work here is excellent and it points where you think it does. A size-correlated truncation that clears the moment you force Accept-Encoding: identity, reproduces across three independent codebases and two SDK versions, and only bites large gzipped bodies is the classic signature of a path-MTU blackhole on the egress path, not your apps or the upstream. We are getting our networking team to look at the US West (California) Metal egress for this window.
To speed up the repro on our side, could you share: a sample failing URL plus the approximate compressed body size where it starts breaking, the exact byte offset or point in the stream where it cuts off if you have it, and whether the truncation is 100 percent consistent for a given large response or intermittent. The service IDs and a couple of full stack traces you offered would help too.
Status changed to Awaiting User Response Railway • 15 days ago
8 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 8 days ago