Skip to content
AWS CloudFront (Brussels) intermittently cannot connect to the Amsterdam edge, about 1 request in 11
studentdave
OP

a day ago

Our production service (housequest-v7-prod, region US West, 4 replicas) sits behind AWS CloudFront. Since at least 17 November 2026 21:48 UTC, some requests that CloudFront's Brussels location (x-amz-cf-pop BRU51-P1) sends to our Railway domain take exactly 10, 20 or 30 seconds longer than normal. At 30 seconds CloudFront gives up and returns a 504. The person who runs our CloudFront distribution confirmed that its configuration has not changed. Requests that do reach the app are answered in under 0.4 seconds.

What we measured from Belgium, 18 November 19:32 UTC to 19 November 06:10 UTC:

  • Through CloudFront Brussels, which reaches your Amsterdam edge (x-railway-edge: ams1): 4,726 requests that CloudFront had to fetch from Railway. 431 were slow (about 1 in 11), and 17 of those failed with a 504 after 30 seconds. Every slow request that returned went through ams1; the few that went through lhr1 were fast.
  • Direct from Brussels to the Railway domain (x-railway-edge: bru1): 4,730 of 4,730 finished in under 3 seconds.
  • Direct from Brussels to the custom domain target 153.52.255.81 with SNI and Host evergreen.housequest.com (x-railway-edge: bru1), 19 November 06:56 UTC: 60 of 60 under 0.7 seconds.
  • Through CloudFront San Francisco (x-amz-cf-pop SFO53-P7), which reaches your Los Angeles edge (x-railway-edge: lax1): 90 of 90 took 0.2 seconds or less (18 November, 20:38 UTC).

One failed request in detail: GET /api/meta, CloudFront request id XlhPD7uotKjzwvCmjOQcYJPevFvNrSTzCAd2A-JwZrj8CQuQnYRhfg==, 504 at 22:32:43 UTC after 30.0 seconds. CloudFront's log for it shows x-edge-detailed-result-type OriginCommError ("the request timed out while connecting to the origin, or reading data from the origin"). Railway's HTTP log for our service has no request from the Brussels CloudFront servers between 22:32:13 and 22:32:44 UTC, so it never reached your proxy. The requests from those servers (srcIp 52.46.47.69 and 52.46.47.108) just before and after it were answered in under 20 milliseconds. In the same 30 seconds, Route 53 health checks reached the service through other CloudFront servers without trouble.

The delays are whole multiples of 10 seconds, which is CloudFront's default origin connection timeout. So CloudFront's Brussels servers fail to open a connection to ams1, wait 10 seconds, and retry, up to three times. Your status page shows the edge network as operational. Can you check whether new connections from AWS address ranges (for example 52.46.47.69 and 52.46.47.108) to ams1 are being dropped or rate limited?

Custom domain target: wxtb4ir8.up.railway.app (153.52.255.81).

$40 Bounty

3 Replies

Railway
BOT

a day ago

We haven't established what's causing this yet. The community can help investigate your setup.

That's exactly what the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public and the thread closes.

Status changed to Awaiting User Response Railway • 1 day ago


Railway
BOT

a day ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 1 day ago


a day ago

This looks like TCP connections failing to connect to ams1 and then not reaching the proxy. Some things you can try as a workaround in CloudFront:

  1. Set the connection timeout from 10s -> 2-3s
  2. Origin keep-alive timeout from 5s -> 60s
  3. Apply an origin shield in eu-west-2 (London), then all fetches go through London, and you showed lhr1 was fast.

Let me know if this helps!


juniordias00

4 hours ago

Hi @studentdave — the 22:32:43 UTC failure is specific enough to escalate to both providers now. I would keep the four US West replicas in place. Your 4,730 direct requests from Brussels reached bru1. So did the separate 60-request test to the same IP with the production Host and SNI. That shows the hostname and service can work through bru1. The slow responses through CloudFront came back through ams1. Railway uses anycast, so even an exact destination IP can lead to a different ingress from a different source network.

Start with the 414 slow requests that actually returned. Of 4,726 origin fetches, 431 were slow (9.12%), but only 17 ended in 504 (0.36%). The other 414 give you a way to investigate the delay on requests where both platforms can have an HTTP record. Compare one fast 200, one 200 delayed by about 10 or 20 seconds, and one 30-second 504 for the same GET path and viewer POP.

AWS defines OriginCommError to include both connection and read timeouts. The 10/20/30-second steps fit the default of three connection attempts at up to ten seconds each. A 30-second origin read timeout can also produce a 504. Confirm the configured values on this origin before interpreting the bands. The missing Railway HTTP entry narrows the investigation, but it cannot tell us whether a SYN reached the edge, a SYN-ACK was lost, or TLS stopped before an HTTP request could be logged. The failed request itself has no Railway response header identifying its ingress.

One measurement from the existing Belgian probe. Ask the distribution owner to enable CloudFront Server-Timing on the response headers policy for the affected behavior. Sampling can stay at 0%. Once enabled, sending Pragma: server-timing opts that request in. Use the same viewer URL your probe already tests, through CloudFront, and keep GET rather than switching to HEAD:

curl -sS -D - -o /dev/null -H 'Pragma: server-timing' \

-w 'HTTP %{http_code} total=%{time_total}s\n' "$VIEWER_URL"

Keep the UTC time, X-Amz-Cf-Id, X-Amz-Cf-Pop, Server-Timing and, when present, X-Railway-Edge and the Railway request ID. Use origin fetches for the comparison. A cache hit can replay stored origin headers. For the slow 200s, three Server-Timing fields matter:

  • cdn-upstream-layer tells you whether the viewer edge, a regional edge cache or Origin Shield contacted the origin. BRU51-P1 identifies the viewer-facing POP. The actual AWS egress for the failed attempt still needs confirmation.
  • cdn-upstream-connect measures origin TCP setup plus TLS for HTTPS. AWS documents zero as connection reuse. A large value alongside a small cdn-upstream-fbl points toward connection setup.
  • cdn-upstream-fbl measures the wait for the first response byte after CloudFront sent the origin request. If that dominates, compare the matching Railway HTTP timings. These headers are not a per-attempt trace, so a quick final connection cannot rule out an earlier failed attempt.

Turn Milo's timeout suggestion into a controlled test. On a staging distribution or a duplicate origin used only by a diagnostic behavior, keep the same origin hostname, Host forwarding, read timeout and attempt count. Change only the connection timeout from 10s to 3s. If the slow bands shift toward 3/6/9s, that supports a connection-attempt timeout. If the 30s band persists, check the read phase and the AWS trace before tuning further. Measure both latency and 504 rate: a shorter wait can also give up on connections that would have succeeded. Test keep-alive or London Origin Shield separately. If you try London, verify which Railway edge answers the origin fetches; selecting an AWS region cannot pin Railway to lhr1.

What I'd ask support to check. Give AWS the distribution ID privately and the full CloudFront request ID already in your post. Ask which layer contacted the origin, which source and destination IPs were used on each attempt, and whether the timeout occurred in TCP, TLS or the HTTP read. Give Railway the Nov 18 22:32:13–22:32:44 UTC window, destination 153.52.255.81:443, and the neighboring successful request IDs. Ask whether available ingress telemetry shows arriving SYNs, replies, TLS failures, or a drop/rate-limit rule. The two AWS source IPs you saw on successes are useful search hints. Matching the actual failed attempt is what would establish where the traffic stopped.

References

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-response-headers-policies.html

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logs-reference.html

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistValuesOrigin.html

https://docs.railway.com/networking/edge-networking

AI-assisted. I checked the documentation. I haven't inspected either provider's internal telemetry.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...