Latency Question: POP -> Region vs Region -> Region
mhornbacher
PROOP

a month ago

Hey everyone. I am looking into a question and having a hard time nailing down a concrete answer. And its a fun question so I thought I would ask everyone...

For some background... As I intend to not be fined under the GDPR for storing a user uploaded picture and the EU-US Data Privacy Framework has been challenged since June it seems easiest to simply store all data in the EU region for now until this political mess is resolved. It is also fast enough for my architecture that it is generally a non issue.

So the fun question. I have:

  1. An IPX server on Bun that optimizes images from a few domains (e.g. google account profile pictures) as well as my internal NextJS service via the internal networking and a Railway Bucket via Bun.s3 (at /public and /file respectively)
  2. A Caddy Reverse Proxy that routes traffic based on the route between the NextJS server, PostHog (EU, Frankfurt), my API server (ElysiaJS) and my Vite SPA for the core offline-first application.

So my question is, which is faster. US-East -> EU internal service traffic, or US POP (say JFK) -> EU traffic. Having a hard time nailing it down but I am curious if this has mattered to anyone at scale?

Solved$20 Bounty

Pinned Solution

a month ago

Chiming in from the Railway side. Right conclusion in this thread, partly wrong reasoning.

Reply 2 has it backwards on the network. Private networking is a L3 WireGuard link on our infra: no TLS, no proxy, no CDN in the path. There's no cold handshake per call unless your app does one. The public edge path (user to CDN to edge proxy to origin) has more hops and routing we don't control. For a single US-East to EU call, private is faster and more consistent.

But Reply 1 is right about what matters: RTT is ~85ms either way, and page latency is set by how many times you cross the Atlantic per request. Everything in EU means one crossing per request. Compute in US-East calling EU services means one crossing per internal call, and IPX pulling per-asset from a bucket or an API doing dozens of queries per page multiplies that fast.

On "why make multi-region easy then": it's for redundancy, and for serving local users with local data.

14 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 1 month ago


larrythiessen
PRO

a month ago

Both options cross the Atlantic. The distance is identical, so the thing that decides this isn't which link is faster, it's how many times you pay the crossing.

Railway's edge is a real terminating proxy, not a pass-through. You can see this in responses (server: railway-hikari, x-railway-edge: atl1), and it's been observed terminating HTTP/2 itself. There's a thread on here documenting the edge cancelling an h2 stream after the origin already returned 200, and closing idle client keep-alives at 60s. Both only happen if the POP terminates the connection.

That distinction is the whole answer:

Everything in EU, US users enter via a POP. The user's TCP + TLS handshake terminates at JFK, a few ms away. Only the request itself rides the long leg to Frankfurt, on a warm, pooled connection. A page load pays the Atlantic roughly once.

Compute in US-East talking to EU internally. Every internal call pays the crossing. US-East to EU is ~80-90ms RTT, and internal calls are chatty by nature. There's another thread on this board right now where someone's pages run 40-90 queries, and at 9ms each that's already 500ms a page. Put those same queries across the Atlantic and it's 40-90 x ~85ms. That's not a slow page, that's a broken one.

So: keep compute next to the data. Since GDPR is already pinning your storage to the EU, pin the services that talk to it there too and let the anycast edge absorb the distance for US users. That's also the cheaper answer operationally, one region to reason about.

The exception, and the only case where the split wins: a US-East service that makes very few EU calls per request and batches them. One or two round trips instead of fifty. Your Vite SPA being offline-first pushes you that way a little, but your IPX server doesn't. Image optimisation pulling from a Railway Bucket is exactly the chatty, per-asset pattern that punishes a split.

To your "has this mattered at scale" question, yes, and it usually shows up exactly as you'd expect: not as one slow request, but as page latency scaling with query count. The fix is almost always co-location rather than tuning the link.


ashu08082003
HOBBY

a month ago

US Edge POP (e.g., JFK) to EU traffic is almost universally faster and more consistent than US-East container to EU internal service traffic. When a US client connects to an Edge POP, the initial TCP and TLS handshakes are terminated locally in approximately 5 to 15 milliseconds. Once connected, the POP proxies the request to your EU origin over a persistent, warm backbone link with optimized BGP routing. In contrast, a cross-region request originating from a US-East container must execute a cold TCP and TLS handshake directly across the Atlantic, incurring an immediate round-trip time (RTT) penalty of roughly 70 to 100 milliseconds before any payload data is transferred. Furthermore, container-to-container inter-region networking on standard cloud backbones is more susceptible to public internet routing variations and tail-latency (p95/p99) spikes.

For your specific Railway architecture—consisting of a Bun IPX server, Caddy reverse proxy, EU PostHog, and S3 bucket storage—storing primary data in the EU keeps you compliant with GDPR considerations while isolating dynamic workloads. To maximize performance globally, place an Anycast Edge CDN layer (such as Cloudflare or Railway’s Edge) in front of your Caddy proxy and apply aggressive cache headers (Cache-Control: public, max-age=31536000, immutable) to your optimized IPX outputs.

Under this setup, an initial uncached request from a US user will terminate its handshake instantly at the JFK Edge POP, fetch and process the raw image from the EU IPX server (~80ms RTT cross-ocean fetch), and cache the output at the edge. Subsequent requests for that image will then be served directly from the JFK Edge POP with a Time to First Byte (TTFB) of 10 to 20 milliseconds, completely bypassing transatlantic latency. Avoid running a US-East compute container solely to proxy internal service calls to EU containers, as it introduces cross-region overhead without the handshake-termination benefits of an Edge POP.


a month ago

Didn't mean to make it more difficult by attaching a PDF, but I just got done creating my own branding PDF generator, so I had to try it. The answer should make sense.

Attachments


mhornbacher
PROOP

a month ago

Thank you guys all so much. It makes a lot of sense that the hot BGP gateway is faster than a cold internal tunnel 😀

I guess I am now wondering why Railway makes it so easy to deploy multi region apps if it will end up being slower… redundancy?


Status changed to Open brody • about 1 month ago


a month ago

Chiming in from the Railway side. Right conclusion in this thread, partly wrong reasoning.

Reply 2 has it backwards on the network. Private networking is a L3 WireGuard link on our infra: no TLS, no proxy, no CDN in the path. There's no cold handshake per call unless your app does one. The public edge path (user to CDN to edge proxy to origin) has more hops and routing we don't control. For a single US-East to EU call, private is faster and more consistent.

But Reply 1 is right about what matters: RTT is ~85ms either way, and page latency is set by how many times you cross the Atlantic per request. Everything in EU means one crossing per request. Compute in US-East calling EU services means one crossing per internal call, and IPX pulling per-asset from a bucket or an API doing dozens of queries per page multiplies that fast.

On "why make multi-region easy then": it's for redundancy, and for serving local users with local data.


Status changed to Awaiting User Response Railway • about 1 month ago


mhornbacher
PROOP

a month ago

Hey brody! Thank you for the great breakdown 🙏

So if I understand correctly, if I had an architecture where users data was stored in a database in their region it would make sense but otherwise relying on the POP routing is the faster/reccomended approach

Is this also true for the Caddy router in my stack or should that be local to the users to use the “faster” T3 tunnel across regions rather than the POP system (it forwards the full request once)


a month ago

The main take away is that you should always use the private network, for anything, it will always be faster.


Status changed to Awaiting User Response Railway • about 1 month ago


mhornbacher
PROOP

a month ago

well unless you are doing multiple round trips across regions 😂

Thank you so much again for such an in depth and useful answer


Status changed to Awaiting Conductor Response Railway • about 1 month ago


moosenberg

Didn't mean to make it more difficult by attaching a PDF, but I just got done creating my own branding PDF generator, so I had to try it. The answer should make sense.

mhornbacher
PROOP

a month ago

Very interesting... I get a 403 looking for the asset. I am sure it is really nice though


mhornbacher

well unless you are doing multiple round trips across regions 😂 Thank you so much again for such an in depth and useful answer

a month ago

Multiple round trips would still be faster via private networking; traffic would make fewer hops and follow a controlled, dedicated path to our other regions.


Status changed to Awaiting User Response Railway • about 1 month ago


mhornbacher

Very interesting... I get a 403 looking for the asset. I am sure it is really nice though

a month ago

Only the user who posts the PDF and admins can open it. This prevents people from sharing invoices and similar documents.


brody

Only the user who posts the PDF and admins can open it. This prevents people from sharing invoices and similar documents.

mhornbacher
PROOP

a month ago

That makes... a lot of sense 😂 Shame. I am sure he put a lot of work into it. I hope you can enjoy it brody ❤️


Status changed to Awaiting Conductor Response Railway • about 1 month ago


a month ago

Truthfully, it's just AI generated text.


Status changed to Awaiting User Response Railway • about 1 month ago


mhornbacher
PROOP

a month ago

That sucks


Status changed to Awaiting Conductor Response Railway • about 1 month ago


Status changed to Solved dev • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...