Does the edge strip client X-Forwarded-For? Two staff answers contradict; our measurement says yes
plotbase-commits
FREEOP

a month ago

Hi, we run an Express app on Railway and needed to configure trust proxy safely for rate limiting. Two Railway staff have answered this publicly in ways that contradict each other, and the difference is security-critical, so we measured it ourselves.

THE CONTRADICTION

phin (EMPLOYEE), in "Security-Critical Questions on Edge Proxy Header Handling and Hop Count": "We do strip X-Forwarded-For at our edge and ensure clients cannot overwrite it. The first value of the X-Forwarded-For is the real connecting IP."

sam-a (EMPLOYEE), in "Which header should I rely on for real client IP?" (thread marked Solved): "clients can send a spoofed X-Forwarded-For, but the real client IP will always be the leftmost entry since our edge proxy appends to the chain."

Those cannot both hold. If the edge appended to a client-supplied chain, the leftmost entry would be the attacker's value, and "take the first IP" would be a spoofing vulnerability rather than the recommendation.

Additionally, in the first thread the accepted answer (a paid bounty answer) states the opposite of phin: "Railway appends the real client IP to the end of XFF. It does NOT strip client-supplied values... Only the rightmost value is trustworthy."

WHAT WE MEASURED

We added a temporary diagnostic to our own health endpoint that logs only boolean verdicts (no IP addresses), then sent requests carrying a sentinel from TEST-NET-3 (RFC 5737) - a range that can never be a real client - plus our own known public address for comparison, via headers X-Forwarded-For: 203.0.113.7, X-Real-IP: 203.0.113.8 and X-Diag-Expect set to our own public address.

Results observed on the receiving side:

  • the sentinel appeared NOWHERE in the chain the app received, and the spoofed X-Real-IP was gone as well; a control request with no headers produced identical output
    • X-Forwarded-For had 2 entries; the leftmost equalled our real public address, the rightmost did not
    • X-Real-IP also equalled our real public address on this path
    • the immediate socket peer was in 100.0.0.0/8; no Fastly-Client-Ip or Cf-Connecting-IP was present

So phin is right: client-supplied X-Forwarded-For does not reach the application, and the leftmost entry is the real client. We also confirmed that trust proxy = 1 reads the rightmost entry, i.e. an infrastructure hop rather than the client - which is what sent us down this path in the first place.

QUESTIONS

  1. Is that stripping behaviour guaranteed and stable, or incidental to the current routing? We would be building rate limiting on it.
  2. Can the contradicting answers be corrected? The sam-a thread is marked Solved and the bounty answer in the other thread is the accepted one, so both currently tell readers the opposite of what we measured - and "only the rightmost value is trustworthy" leads directly to keying rate limits on a shared CDN hop instead of on the client.
  3. During the CDN rollout, is X-Real-Ip still expected to be set to the CDN edge IP rather than the client (the bug sam-a mentioned)? On our path it was correct, so we cannot tell whether it is fixed or whether we simply were not on the affected route. Is there a target date?
  4. Is the immediate peer range (100.0.0.0/8) something we may rely on for an Express trust proxy function, or is it an implementation detail?

We are not asking for a change - we just want to know which of these we may treat as a contract.

Thanks!

$10 Bounty

2 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 27 days ago


X-Real-IP is written at the edge, so it can’t be spoofed/will be overwritten. X-Forwarded-For is also overwritten. And yes, your Express server should trust that range, as requests could come from within that range from Railway’s internal servers.


plotbase-commits
FREEOP

a month ago

Thanks — that helps, and the X-Forwarded-For part matches what we measured: client-supplied XFF never reached our app.

On the trust proxy advice I want to flag something for anyone else who lands here, because it does not hold on a CDN-routed path.

Our observed chain is [ real client , 152.233.x ], where the rightmost hop is CDN77 / Datacamp (AS60068) - not in 100.0.0.0/8. proxy-addr walks the chain right to left and stops at the first untrusted address, so trusting only 100.0.0.0/8 stops at the CDN hop and returns it. Measured against that exact chain with Express:

  • trust 100.0.0.0/8 only -> CDN hop
    • trust proxy = 1 -> CDN hop
    • trust 100.0.0.0/8 AND 152.233.0.0/16 -> real client

So on that route the recommendation reproduces the original problem, and making it work would mean trusting a third-party CDN's ranges, which is what we were trying to avoid. We are going with reading the leftmost XFF entry instead and leaving trust proxy alone, since the leftmost is not spoofable given the edge overwrites the header.

Still hoping for a staff answer on three things:

  1. Is the overwriting guaranteed and stable, or incidental to the current routing? That is the part we would be building rate limiting on.

  2. X-Real-Ip: "cannot be spoofed" is not the same as "is the real client". sam-a stated that when traffic passes through the CDN layer, X-Real-Ip is set to the CDN edge IP rather than the true client, and called it a bug being tracked. Is that still the case, and is there a target date? On our path X-Real-IP was correct, so we cannot tell whether it is fixed or whether we simply were not on the affected route.

  3. Can the two contradicting threads be corrected? One is marked Solved, and the other's accepted bounty answer says "only the rightmost value is trustworthy", which leads readers straight to keying rate limits on a shared CDN hop.

Thanks!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...