2 months ago
Service: Node.js Express + Apollo GraphQL
Issue: Outbound HTTPS streams to AWS S3 stall mid-transfer approximately 1/300 requests
What happens:
GetObjectCommandis sent to S3 — succeeds, headers received with HTTP 200- Stream body begins transferring (~100kb file)
- Stream silently stalls mid-drain — no error, no RST, no FIN, just stops delivering data
- Hangs indefinitely until something kills the connection (~4 minutes)
What we've ruled out:
- Not an S3 issue — never happens when the same code runs locally
- Not a code issue — headers arrive fine, only the body stalls
- Not a retry issue — the AWS SDK sees a 200 and considers the request successful, so it never retries
- Only affects S3 (outbound HTTPS streaming) — Postgres and Redis connections on the same service are stable
Key detail:
The failure is completely silent — no error event is raised on the stream, no exception is thrown. The stream just stops emitting data events mid-transfer and never emits end. This suggests something in Railway's network layer is dropping packets mid-stream without closing the connection, rather than actively terminating it.
Suspected cause:
Railway's egress network (NAT gateway or outbound proxy) is silently dropping packets mid-stream on outbound HTTPS connections approximately 1/300 times. Because no RST or FIN is sent, the TCP connection appears open but data stops flowing.
Questions:
- Is there a known idle or mid-stream timeout on outbound connections from Railway containers?
- Is outbound traffic routed through a proxy or NAT gateway that could be dropping packets?
- Are there any known issues with outbound HTTPS streaming to AWS S3 specifically?
1 Replies
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
Thanks for the detailed writeup, that rules out most of the usual suspects up front.
To answer your three questions:
-
There is no application-level idle or mid-stream timeout that Railway imposes on outbound connections. We do not cap how long an outbound stream can run.
-
Yes. Outbound traffic from containers egresses through a shared NAT layer. A silent mid-stream stall with no RST/FIN is consistent with a conntrack/SNAT edge case on long-lived streaming flows rather than an active termination, which lines up with your observation that the socket stays open but data stops. This is the layer worth investigating, and I am routing the specifics to our networking team.
A few things that will help us pin it down, and that also tend to mitigate it in practice:
- Enable TCP keepalives on the S3 client socket (low interval, e.g. 15s). Keepalives keep the conntrack entry warm and surface a dead path as a real error instead of an indefinite hang.
- Set an explicit per-request/socket timeout in the AWS SDK (requestTimeout / socket idle timeout) so a stalled body fails fast and the SDK retries instead of waiting ~4 minutes. Since the SDK already saw a 200, a socket-level timeout is what converts this into a retryable failure.
- Static outbound IPs route egress through a different path and can behave differently for long-lived flows, so it is worth testing against your 1/300 rate. Note these are shared, not dedicated. I would pair it with the keepalive/timeout changes above rather than rely on it alone.
If you can grab the offending five-tuple (source/dest IP:port) and a rough timestamp the next time it stalls, send it over and we can correlate it against the egress path directly.
— Angelo
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 2 months ago