Outbound HTTPS streams to AWS S3 stall mid-transfer approximately 1/300 requests
matthewsmkim1
PROOP

2 months ago

Service: Node.js Express + Apollo GraphQL

Issue: Outbound HTTPS streams to AWS S3 stall mid-transfer approximately 1/300 requests

What happens:

  1. GetObjectCommand is sent to S3 — succeeds, headers received with HTTP 200
  2. Stream body begins transferring (~100kb file)
  3. Stream silently stalls mid-drain — no error, no RST, no FIN, just stops delivering data
  4. Hangs indefinitely until something kills the connection (~4 minutes)

What we've ruled out:

  • Not an S3 issue — never happens when the same code runs locally
  • Not a code issue — headers arrive fine, only the body stalls
  • Not a retry issue — the AWS SDK sees a 200 and considers the request successful, so it never retries
  • Only affects S3 (outbound HTTPS streaming) — Postgres and Redis connections on the same service are stable

Key detail:

The failure is completely silent — no error event is raised on the stream, no exception is thrown. The stream just stops emitting data events mid-transfer and never emits end. This suggests something in Railway's network layer is dropping packets mid-stream without closing the connection, rather than actively terminating it.

Suspected cause:

Railway's egress network (NAT gateway or outbound proxy) is silently dropping packets mid-stream on outbound HTTPS connections approximately 1/300 times. Because no RST or FIN is sent, the TCP connection appears open but data stops flowing.

Questions:

  1. Is there a known idle or mid-stream timeout on outbound connections from Railway containers?
  2. Is outbound traffic routed through a proxy or NAT gateway that could be dropping packets?
  3. Are there any known issues with outbound HTTPS streaming to AWS S3 specifically?
Solved

1 Replies

Status changed to Awaiting Railway Response Railway about 2 months ago


Thanks for the detailed writeup, that rules out most of the usual suspects up front.

To answer your three questions:

  1. There is no application-level idle or mid-stream timeout that Railway imposes on outbound connections. We do not cap how long an outbound stream can run.

  2. Yes. Outbound traffic from containers egresses through a shared NAT layer. A silent mid-stream stall with no RST/FIN is consistent with a conntrack/SNAT edge case on long-lived streaming flows rather than an active termination, which lines up with your observation that the socket stays open but data stops. This is the layer worth investigating, and I am routing the specifics to our networking team.

A few things that will help us pin it down, and that also tend to mitigate it in practice:

  • Enable TCP keepalives on the S3 client socket (low interval, e.g. 15s). Keepalives keep the conntrack entry warm and surface a dead path as a real error instead of an indefinite hang.
  • Set an explicit per-request/socket timeout in the AWS SDK (requestTimeout / socket idle timeout) so a stalled body fails fast and the SDK retries instead of waiting ~4 minutes. Since the SDK already saw a 200, a socket-level timeout is what converts this into a retryable failure.
  • Static outbound IPs route egress through a different path and can behave differently for long-lived flows, so it is worth testing against your 1/300 rate. Note these are shared, not dedicated. I would pair it with the keepalive/timeout changes above rather than rely on it alone.

If you can grab the offending five-tuple (source/dest IP:port) and a rough timestamp the next time it stalls, send it over and we can correlate it against the egress path directly.

— Angelo


Status changed to Awaiting User Response Railway about 2 months ago


Railway
BOT

2 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...