Hawaii traffic seems to route through Sao Paulo
18thletter
HOBBYOP

a month ago

I'm in Hawaii (Hawaii island), and my traffic seems to be getting routed through Brazil on every request.

From Hawaiian Telcom in Hawaii, every Railway-hosted property answers from the gru1 (Sao Paulo) edge -- including railway.com and docs.railway.com, so it isn't specific to my services. My services are in us-west2, so requests cross to Brazil and hairpin back to Oregon, adding ~1s to every request.

[myhost] / railway.com / docs.railway.com -> x-railway-edge: gru1

RTT from the same host (so not my last mile):

69.46.46.39 (Railway anycast)    243.1 ms

1.1.1.1     (Cloudflare)                  13.0 ms

8.8.8.8     (Google)                       56.2 ms

Path — Lumen picks it up locally, then hauls it to Brazil:

2  72.253.4.3     4.9 ms   Hawaiian Telcom

4  4.79.8.25     12.4 ms   AS3356 Lumen, Honolulu

6  8.242.0.2    243.0 ms   AS3356 Lumen, São Paulo

Traffic is already on Lumen in Honolulu at ~12ms, then travels ~230ms to Sao Paulo to reach AS400940. I checked other regions and saw normal figures (US 0.12 - 0.18s, EU 0.26 - 0.34s, BR 0.47s); Hawaii is the outlier. IP is visible to 323/326 RIS peers, so propagation is fine.

Fronting with Cloudflare only fixes half -- client -> edge drops to ~12ms (their Honolulu PoP) but the response still says x-railway-edge: gru1 and total only falls 1.29s -> 0.51s, because their backbone also lands on gru1.

Anyone know a workaround or fix?

Solved

11 Replies

Railway
BOT

a month ago

Your diagnostics show the path from Hawaiian Telcom through Lumen (AS3356) reaching Railway via São Paulo rather than a US mainland path, which is driven by how your transit provider's BGP routing selects a path to the destination. This is not specific to your services or anything configurable on your side. Cloudflare in front partially helps because it terminates the client connection at its Honolulu PoP, but the origin fetch still follows a similar transit path, explaining the partial improvement you measured.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway

Your diagnostics show the path from Hawaiian Telcom through Lumen (AS3356) reaching Railway via São Paulo rather than a US mainland path, which is driven by how your transit provider's BGP routing selects a path to the destination. This is not specific to your services or anything configurable on your side. Cloudflare in front partially helps because it terminates the client connection at its Honolulu PoP, but the origin fetch still follows a similar transit path, explaining the partial improvement you measured.

18thletter
HOBBYOP

a month ago

Yep, you got it right. Still need help on this.


Status changed to Awaiting Railway Response Railway • about 1 month ago


a month ago

Thank you for your report, we are looking into this.


Status changed to Awaiting User Response Railway • about 1 month ago


a month ago

Hello,

We've withdrawn from a global transit provider (Lumen) in Sao Paulo which has now fixed this issue. Thank you for the report!


a month ago

We have made some changes to our routing table, can you confirm if this is still ongoing?


brody

We have made some changes to our routing table, can you confirm if this is still ongoing?

18thletter
HOBBYOP

a month ago

Path (tracepath -n, annotated with reverse DNS):

hop  ip                reverse dns                 location                             rtt
---  ----------------  --------------------------  ------------------------------  --------
  2  72.253.4.3        (none)                      Hawaiian Telcom, Hilo HI          4.9 ms
  3  *                 *                           *                                      *
  4  72.52.108.226     e0-38.core3.sjc2.he.net     Hurricane Electric, San Jose      64.8 ms
  5  184.104.189.61    be2.core1.nyc4.he.net       Hurricane Electric, New York     193.8 ms
  6  184.104.196.70    be10.core4.lon2.he.net      Hurricane Electric, London       197.9 ms
  7  185.1.104.29      60068-ld7-ix.equinix.com    Equinix IX London (LD7)          191.9 ms

Now it’s going through London instead of Brazil. So still looks like anycast could use some adjustment for this specific route (or probably all routes around hawaii). Looks like hawaii Telcom will hand off cleanly to hurricane electric in San Jose. But anything on west coast would be better. Somehow London is winning now.


Status changed to Awaiting Railway Response Railway • about 1 month ago


a month ago

We have made additional changes. If you still don't see any improvements, could you please provide an up-to-date traceroute?


Status changed to Awaiting User Response Railway • about 1 month ago


18thletter
HOBBYOP

a month ago

Thanks for making the change, but seems like nothing moved.

Sampled every 20 minutes for almost a day:

samples                73
errored samples         0
x-railway-edge         lhr1 on 73 of 73 samples
railway rtt (ms)       min 206.566   median 206.818   max 208.195
ttfb /health (s)       min   0.769   median   0.856   max   0.941

Control, same host, same 20-minute cadence, to a Cloudflare-fronted name:

cloudflare rtt (ms)    min  12.040   median  12.594   max  13.806

The control stayed flat for the whole window, so the local link was healthy

throughout and the ~207 ms is entirely the path to the Railway edge.

Current path, unchanged from the previous report:

hop  ip                reverse dns                 location                             rtt
---  ----------------  --------------------------  ------------------------------  --------
  2  72.253.4.3        (none)                      Hawaiian Telcom, Hilo HI          4.9 ms
  3  *                 *                           *                                      *
  4  72.52.108.226     e0-38.core3.sjc2.he.net     Hurricane Electric, San Jose      73.3 ms
  5  184.104.189.61    be2.core1.nyc4.he.net       Hurricane Electric, New York     194.5 ms
  6  184.104.196.70    be10.core4.lon2.he.net      Hurricane Electric, London       198.1 ms
  7  185.1.104.29      60068-ld7-ix.equinix.com    Equinix IX London (LD7)          192.1 ms

Status changed to Awaiting Railway Response Railway • 29 days ago


sam-a
EMPLOYEE

a month ago

Thanks for the detailed follow-up. Your latest results show that our changes haven’t resolved the London routing for your connection. I’ve passed these results to our networking team for further review. I’m sorry this is still dragging on. I don’t have a fix or an ETA to share yet, but I’ll update you here when I have more information.


Status changed to Awaiting User Response Railway • 29 days ago


a month ago

We added additional routing fixes. Are you still seeing incorrect routing on your end?


18thletter
HOBBYOP

a month ago

Fixed, thank you!

Edge moved to lax1:

2026-09-17T10:40Z   cdg1 -> lax1     61.4 ms   (stable since)

Before and after:

             rtt          ttfb /health
lhr1    206.804 ms          0.860 s     (n=205)
lax1     61.779 ms          0.312 s     (n=14)

3x faster

Appreciate you guys sticking with it!


Status changed to Awaiting Railway Response Railway • 27 days ago


Status changed to Solved 18thletter • 27 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...