Requests are not served from correct edge
arjunkomath
PROOP

10 months ago

I've a web server deployed to all regions and I'm testing requests from Sydney. I expected the response to be served from Singapore as its the closed, but it seems to be served from us east which more that double the distance compared to Singapore.

Solved

25 Replies

ray-chen
EMPLOYEE

10 months ago

Hey!

Could you share the response of https://rlwy-edge-info-production.up.railway.app/ and traceroute 66.33.22.11 please?

Status changed to Awaiting User Response Railway • 10 months ago

arjunkomath
PROOP

10 months ago

Edge info

edge=railway/us-east4-eqdc4a zone=us-east4-eqdc4a ip=::ffff:100.64.0.4
forwarded=[redacted] hs=5cf4fc755882 req=OO21JlN-TY-SoaD6ZJ9Vjg_28081791

Status changed to Awaiting Railway Response Railway • 10 months ago

arjunkomath
PROOP

10 months ago

traceroute to 66.33.22.11 (66.33.22.11), 64 hops max, 40 byte packets
 1  unifi (192.168.1.1)  5.009 ms *  6.674 ms
 2  100.127.0.193 (100.127.0.193)  5.219 ms  13.695 ms  9.519 ms
 3  vl102.pe02.syd01.launtel.net.au (103.214.222.33)  6.326 ms  6.832 ms  6.912 ms
 4  vl100.pe01.syd01.launtel.net.au (103.214.222.156)  7.755 ms  5.314 ms  7.004 ms
 5  unknown.globalsecurelayer.com (103.137.13.26)  5.675 ms  5.406 ms  5.143 ms
 6  po8.syd-eqxsy5-bb1.globalsecurelayer.com (206.148.24.112)  6.109 ms  5.878 ms  5.148 ms
 7  e52.lax-csla2-bb1.globalsecurelayer.com (206.148.24.3)  138.334 ms  138.770 ms  138.779 ms
 8  e7.chi-eqxch1-bb1.globalsecurelayer.com (206.148.25.78)  177.167 ms  176.712 ms  177.687 ms
 9  e50.pit-dbpit1-bb1.globalsecurelayer.com (206.148.25.65)  190.607 ms  195.283 ms  190.315 ms
10  e58.ash-eqxdc10-bb3.globalsecurelayer.com (206.148.25.62)  193.256 ms  193.499 ms  193.943 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
brody
EMPLOYEE

10 months ago

Do you happen to be on any kind of VPN? maybe even Apple's Private Relay?

Status changed to Awaiting User Response Railway • 10 months ago

arjunkomath
PROOP

10 months ago

Private relay is off, and no VPN. Also, I can see my IP address in https://rlwy-edge-info-production.up.railway.app/ response.

Status changed to Awaiting Railway Response Railway • 10 months ago

arjunkomath
PROOP

10 months ago

I tried the edge info on my phone with mobile data, its going to us-west2:

edge=railway/us-west2
zone=us-west2
ip=::ffff:100.64.0.2
forwarded=1.129.[...]
hs=5f31e3245d19
req=u_ezqV04SQu2Llpf7753NQ_2660894097

But, when I try a VPN in Australia is routed to Singapore as expected grinning emoji , this is using VPN:

edge=railway/asia-southeast1-eqsg3a
zone=asia-southeast1-eqsg3a
ip=::ffff:100.64.0.2
forwarded=37.120.234.154
hs=9977d63d8956
req=vckwnA0QTaW4VoZjx4qX6Q_1774336823

So, its seems like my home WiFi and mobile data isn't working, but VPN is.

Railway
BOT

10 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway • 10 months ago

ray-chen
EMPLOYEE

10 months ago

Thank you for sharing the above! That is odd.

We're going to need a little bit more information to dig into this:

  • Your ISP (the one that gets resolved to US East)

  • Your mobile data provider (the one that gets resolved to US West)

  • A traceroute from your mobile data, if possible (via hotspot/tethering)

  • Details of your VPN provider (connect to VPN -> paste contents of https://ipinfo.io/json here)

Thanks!

ray-chen

Thank you for sharing the above! That is odd.We're going to need a little bit more information to dig into this:Your ISP (the one that gets resolved to US East)Your mobile data provider (the one that gets resolved to US West)A traceroute from your mobile data, if possible (via hotspot/tethering)Details of your VPN provider (connect to VPN -> paste contents of https://ipinfo.io/json here)Thanks!

arjunkomath
PROOP

10 months ago

traceroute on mobile data (Telstra):

traceroute to 66.33.22.11 (66.33.22.11), 64 hops max, 40 byte packets
 1  172.20.10.1 (172.20.10.1)  3.744 ms  5.160 ms  8.171 ms
 2  * * *
 3  10.4.29.8 (10.4.29.8)  75.173 ms
    10.4.28.8 (10.4.28.8)  38.333 ms
    10.4.29.8 (10.4.29.8)  42.821 ms
 4  bundle-ether52.chw-edge903.sydney.telstra.net (203.54.181.189)  40.015 ms  41.541 ms  41.651 ms
 5  bundle-ether12.stl-core30.sydney.telstra.net (203.50.11.176)  56.583 ms  39.047 ms  47.854 ms
 6  * bundle-ether2.pad-gw30.sydney.telstra.net (203.50.6.116)  30.042 ms  38.769 ms
 7  bundle-ether1.sydp-core03.telstraglobal.net (203.50.13.86)  40.093 ms  119.132 ms  20.747 ms
 8  bundle-ether1.sydp-core03.telstraglobal.net (203.50.13.86)  48.451 ms  81.099 ms  108.849 ms
 9  i-10201.sydp-core04.telstraglobal.net (202.84.222.134)  40.413 ms  60.069 ms  30.942 ms
10  i-10201.sydp-core04.telstraglobal.net (202.84.222.134)  169.983 ms  174.284 ms  162.732 ms
11  i-20802.eqnx-core02.telstraglobal.net (202.84.141.25)  153.711 ms  622.162 ms  323.566 ms
12  i-92.eqnx03.telstraglobal.net (202.84.247.17)  310.482 ms  262.985 ms  190.615 ms
13  ae10.edge1.sjo1.sp.lumen.tech (4.68.68.125)  422.135 ms  504.966 ms  410.685 ms
14  * ae2.3605.edge9.sanjose1.net.lumen.tech (4.69.219.65)  350.039 ms *
15  63.215.207.218 (63.215.207.218)  430.788 ms  403.768 ms  407.633 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

ip info on VPN (https://clearvpn.com/)

{
  "ip": "37.120.234.158",
  "hostname": "no-rdns.m247.com",
  "city": "Sydney",
  "region": "New South Wales",
  "country": "AU",
  "loc": "-33.9260,151.1935",
  "org": "AS9009 M247 Europe SRL",
  "postal": "2020",
  "timezone": "Australia/Sydney",
  "readme": "https://ipinfo.io/missingauth"
}

traceroute on VPN:

traceroute to 66.33.22.11 (66.33.22.11), 64 hops max, 40 byte packets
 1  110.110.1.2 (110.110.1.2)  6.964 ms  7.674 ms  6.781 ms
 2  110.110.1.1 (110.110.1.1)  5.981 ms  6.613 ms  6.398 ms
 3  vlan179.as01.syd2.au.m247.ro (37.120.234.153)  7.678 ms  6.631 ms  6.790 ms
 4  irb-1901.agg1v.syd2.au.m247.ro (193.27.15.244)  22.891 ms  40.551 ms  21.335 ms
 5  be-1-3902.bb2.syd1.au.m247.ro (77.243.185.66)  9.836 ms  6.718 ms  6.972 ms
 6  unknown.telstraglobal.net (210.176.152.177)  9.764 ms
    113.29.24.1 (113.29.24.1)  27.028 ms  26.081 ms
 7  i-91.sydo-core04.telstraglobal.net (202.84.222.197)  9.989 ms  8.914 ms  9.261 ms
 8  unknown.telstraglobal.net (210.57.53.60)  96.561 ms
    i-91.sydo-core04.telstraglobal.net (202.84.222.197)  146.290 ms
    unknown.telstraglobal.net (210.57.53.60)  97.722 ms
 9  i-10604.1wlt-core02.telstraglobal.net (202.84.141.225)  145.573 ms  145.221 ms  146.124 ms
10  * i-10604.1wlt-core02.telstraglobal.net (202.84.141.225)  158.714 ms *
11  * i-93.tlot02.telstraglobal.net (202.84.253.86)  144.269 ms  144.073 ms
12  134.159.61.106 (134.159.61.106)  151.495 ms * *
13  * * *
14  * 63.215.207.218 (63.215.207.218)  150.380 ms *
15  * * *
16  * * *

Status changed to Awaiting Railway Response Railway • 10 months ago

ray-chen
EMPLOYEE

10 months ago

Thank you! I'll keep you posted as we dig into this slightly_smiling_face emoji

+11

Status changed to Awaiting User Response Railway • 10 months ago

Railway
BOT

10 months ago

🛠️ The internal ticket Metal edge from Sydney is resolving to US East has been marked as todo.

phin
EMPLOYEE

10 months ago

Hello!

We have made some routing configuration changes on our end. The tier 2 transit provider (Global Secure Layer) which erroneously routed to US East now takes a path to Singapore:

Please let me know if this is fixed from your end.

Best

Phin

Attachments

image.png
heart1
arjunkomath
PROOP

10 months ago

Thank you! the issue has been resolved, I can see all requests are served from asia-southeast1-eqsg3a

Status changed to Awaiting Railway Response Railway • 10 months ago

Status changed to Solved arjunkomath • 10 months ago

Railway
BOT

10 months ago

✅ The internal ticket Metal edge from Sydney is resolving to US East has been marked as completed.

arjunkomath
PROOP

4 months ago

Hey, this is broken again. Requests from Sydney is going to us-west2

Response from https://rlwy-edge-info-production.up.railway.app , I tried on both WiFi and mobile data.

edge=railway/us-west2
zone=us-west2
ip=::ffff:100.64.0.5
forwarded=xxx.xxx
hs=d2b867553204
req=QyRg6ybZT6K39S2S-_9nXA

I also tested from an AU VPN, and it works fine, response:

edge=railway/asia-southeast1-eqsg3a
zone=asia-southeast1-eqsg3a
ip=::ffff:100.64.0.2
forwarded=37.120.234.154
hs=319ddfd1f46a
req=JxKKTWfbTt6YtWRgDcO5xA

Status changed to Awaiting Railway Response Railway • 4 months ago

Status changed to Solved arjunkomath • 4 months ago

arjunkomath
PROOP

4 months ago

Sorry, reopening as I accidentally resolved it

Status changed to Awaiting Railway Response Railway • 4 months ago

arjunkomath
PROOP

4 months ago

Here is my ISP info also, in case you need it

{
  "ip": "2xxxxxxx8",
  "hostname": "118.193.233.220.static.exetel.com.au",
  "city": "Sydney",
  "region": "New South Wales",
  "country": "AU",
  "loc": "xxxxxxxxx",
  "org": "AS10143 Exetel Pty Ltd",
  "postal": "1001",
  "timezone": "Australia/Sydney",
  "readme": "https://ipinfo.io/missingauth"
}
Railway
BOT

4 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway • 4 months ago

phin
EMPLOYEE

4 months ago

Please could you run a traceroute/mtr to 66.33.22.1 and show me the results after running it for ~20 seconds?

phin

Please could you run a traceroute/mtr to 66.33.22.1 and show me the results after running it for ~20 seconds?

arjunkomath
PROOP

4 months ago

Here you go

On my ISP (which routes to us west),

$ timeout 20 traceroute 66.33.22.1
traceroute to 66.33.22.1 (66.33.22.1), 64 hops max, 40 byte packets
 1  unifi (192.168.1.1)  7.566 ms  2.966 ms  2.758 ms
 2  17.1.233.220.static.exetel.com.au (220.233.1.17)  5.365 ms  5.384 ms  5.090 ms
 3  10.1.26.81 (10.1.26.81)  6.181 ms  7.320 ms  6.339 ms
 4  10.1.21.197 (10.1.21.197)  5.269 ms  5.496 ms  5.285 ms
 5  10.1.21.198 (10.1.21.198)  6.507 ms  6.886 ms  7.473 ms
 6  10.1.26.73 (10.1.26.73)  5.673 ms
    10.1.26.63 (10.1.26.63)  5.938 ms
    10.1.26.73 (10.1.26.73)  5.626 ms
 7  bundle-ether200.bdr03-ipt-47bourke-syd.au.as38195.net (116.255.21.108)  9.291 ms  6.857 ms  8.279 ms
 8  hundredgige0-0-1-2.921.bdr01-ipt-624sgran-lax.us.as38195.net (103.200.13.168)  173.947 ms  175.680 ms  174.144 ms
 9  lax-b22-link.ip.twelve99.net (62.115.168.116)  174.324 ms  174.345 ms  173.974 ms
10  lax-bb1-link.ip.twelve99.net (62.115.140.226)  153.076 ms
    lax-bb2-link.ip.twelve99.net (62.115.140.156)  153.469 ms
    lax-bb1-link.ip.twelve99.net (62.115.140.226)  153.118 ms
11  palo-bb2-link.ip.twelve99.net (62.115.137.206)  164.23 *ms  162.797 ms
12  * * sjo-b23-link.ip.twelve99.net (62.115.139.117)  188.859 ms
13  railwaycorp-ic-390075.ip.twelve99-cust.net (62.115.196.225)  157.620 ms  157.492 ms  159.672 ms

on VPN (which routes to Singapore as expected),

$ timeout 20 traceroute 66.33.22.1
traceroute to 66.33.22.1 (66.33.22.1), 64 hops max, 40 byte packets
 1  110.110.7.114 (110.110.7.114)  8.462 ms  6.727 ms  6.869 ms
 2  110.110.7.113 (110.110.7.113)  7.255 ms  6.731 ms  7.425 ms
 3  vlan179.as01.syd2.au.m247.ro (37.120.234.153)  6.709 ms  7.826 ms  7.273 ms
 4  irb-1901.agg1v.syd2.au.m247.ro (193.27.15.244)  34.597 ms  25.716 ms  103.845 ms
 5  te-0-0-0-44.bb1n.lon1.uk.m247.ro (77.243.185.56)  8.009 ms
    be-1-3902.bb2.syd1.au.m247.ro (77.243.185.66)  7.593 ms
    te-0-0-0-44.bb1n.lon1.uk.m247.ro (77.243.185.56)  7.263 ms
 6  113.29.24.1 (113.29.24.1)  8.989 ms  17.949 ms
    unknown.telstraglobal.net (210.176.152.177)  8.863 ms
 7  ae2.3605.edge3.singapore3.level3.net (4.69.206.178)  99.784 ms  100.354 ms  98.754 ms
 8  i-91.sydo-core04.telstraglobal.net (202.84.222.197)  146.717 ms  147.225 ms  147.038 ms
 9  unknown.telstraglobal.net (202.127.73.231)  97.127 ms  127.961 ms
    i-10604.1wlt-core02.telstraglobal.net (202.84.141.225)  145.829 ms
10  i-10604.1wlt-core02.telstraglobal.net (202.84.141.225)  145.691 ms  228.662 ms  145.911 ms
11  * i-95.eqla02.telstraglobal.net (202.84.143.30)  145.255 ms  145.362 ms
12  * *

Status changed to Awaiting Railway Response Railway • 4 months ago

angelo-railway
EMPLOYEE

4 months ago

Hey Arjun, just an update, we're waiting on the Infra team to just double check your traceroute so you can get final resolution. Apologies on the delay.

+11

Status changed to Awaiting User Response Railway • 4 months ago

angelo-railway

Hey Arjun, just an update, we're waiting on the Infra team to just double check your traceroute so you can get final resolution. Apologies on the delay.

arjunkomath
PROOP

4 months ago

No worries.

Status changed to Awaiting Railway Response Railway • 4 months ago

phin
EMPLOYEE

4 months ago

Hello,

I've reached out to the backbone ISP at fault (Superloop AS38195) to request peering with them at a mutual internet exchange in Singapore, so that we can send them preferential routes. I'll update you once we've established peering.

+11

Status changed to Awaiting User Response Railway • 4 months ago

Railway
BOT

4 months ago

🛠️ The ticket Routing issue with Sydney requests has been marked as todo.

phin
EMPLOYEE

4 months ago

The backbone ISP Superloop has just established a peering session with us in Singapore, so that route should now be preferred by them:

Please let me know if this is now resolved.

Attachments

image.png
arjunkomath
PROOP

4 months ago

Yes, it’s fixed now. Thank you.

Status changed to Awaiting Railway Response Railway • 4 months ago

Status changed to Solved arjunkomath • 4 months ago

Railway
BOT

4 months ago

✅ The ticket Routing issue with Sydney requests has been marked as completed.

Loading...