a year ago
I've noticed an issue with extremely high latency between services on Railway within the same region (singapore) when using Metal.
I have a backend service and a frontend service. When using Metal the API calls from the frontend to the backend are taking 1000-2000ms each. The logs of the backend are reporting responses are taking 7-14ms but the frontend is reporting extremely high latency.
Once I switched to the GCP (non-metal) deployment instead then this extreme latency goes away.
This is a very concerning issue that is preventing me from migrating our production environment from AWS to Railway. There should be no reason an API call from Singapore -> Singapore even if happening from Railway Metal -> GCP should be taking 1000-2000ms.
13 Replies
a year ago
May I ask if you're experiencing the latency between Metal<->GCP or Metal<->Metal?
a year ago
Also, are these requests going over the public or private network?
It was Metal<->Metal and over the public network. When switching back to GCP<->GCP the latency issue dissapears.
a year ago
Hey @MrK, could you paste the output of:
traceroute 66.33.22.11
in here?
We have some routing inconsistencies between APAC ISPs to our metal edge network that we're working on fixing.
edit:
To clarify, are those requests to your backend happening over the browser or are you making them from frontend->backend directly (e.g. SSR)?
edge=railway/us-east4-eqdc4a
zone=us-east4-eqdc4a
ip=::ffff:100.64.0.2
forwarded=61.91.45.126
hs=5cf4fc755882
req=ewo1D0nBSy-PnbaJa4hcJg_2206645505traceroute 66.33.22.11
traceroute to 66.33.22.11 (66.33.22.11), 64 hops max, 40 byte packets
1 172.16.10.1 (172.16.10.1) 7.068 ms 4.308 ms 2.660 ms
2 61-91-45-125.static.asianet.co.th (61.91.45.125) 4.955 ms 4.357 ms 5.264 ms
3 171-102-248-111.static.asianet.co.th (171.102.248.111) 5.563 ms 6.338 ms 5.599 ms
4 171-102-248-54.static.asianet.co.th (171.102.248.54) 14.874 ms
171-102-248-110.static.asianet.co.th (171.102.248.110) 5.507 ms 6.582 ms
5 171-102-250-1.static.asianet.co.th (171.102.250.1) 5.991 ms 5.612 ms 6.190 ms
6 * * *
7 171-102-251-244.static.asianet.co.th (171.102.251.244) 6.902 ms 6.437 ms *
8 tig-net28-92.trueintergateway.com (122.144.28.92) 7.216 ms
tig-net28-90.trueintergateway.com (122.144.28.90) 5.839 ms
tig-net28-10.trueintergateway.com (122.144.28.10) 6.517 ms
9 tig-net25-33.trueintergateway.com (122.144.25.33) 30.790 ms
sg-icr-gs2-241-197.trueintergateway.com (113.21.241.197) 34.054 ms *
10 * * e0-38.core3.sin1.he.net (184.104.208.92) 33.082 ms
11 * * *
12 * * *The latency is not via a call from the browser. It's from a call via the frontend service -> backend service. We are using Remix and are fetching data via a loader (SSR).
Remix Loader -> Calls Backend API -> Renders HTML to web client
Latency is occuring within the Remix loader call.
Here is some output from the HTTP logs in the Railway dashboard. This particular request took 2.6 seconds, but the backend logs reports the response was generated in 18ms
```
requestId:
"Wsjiy6aNQj-56RzAHVbYhA98031763" timestamp: "2025-03-19T10:23:23.751832195Z" method: "GET" path: "/my-profile" host: "" httpStatus: 200 upstreamProto: "HTTP/1.1" downstreamProto: "HTTP/2.0" responseDetails: "" totalDuration: 2650 upstreamAddress: "http://[fd12:1c2f:bba7:0:9000:9:ba4:aa09]:8080" clientUa: "Mozilla/5.0 (Macintosh; Intel Mac OS X 1015_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
upstreamRqDuration:
2650
txBytes:
37997
rxBytes:
3151
srcIp:
"
I too seem to be having increased loading times between GCP and Metal.
My Postgress DB is still GCP and when I put other 2 services on Metal users start complaining about long load times. The loadtimes are normal when everything is on GCP (Oregon).
a year ago
Hello, specifically for the US West geography, Railway Metal and Railway GCP are located in two entirely different regions. (GCP Oregon & Railway California). Because of this, you may experience about 15-20ms of latency between these two regions. If you have many round-trips to your database before loading a resource, then that could be why this issue is noticable to your users.
Please note that this issue is specific to the US West geography. All other Railway regions are physically located adjacent to or nearby their GCP regions.
Alright. I had been waiting to move my Postgress service to metal, but the option was always not there. Now I made a new Postgress service and the new one can be metal. I will be moving everything to California Metal.
Any updates on this? The level of support being offered here when paying for a Pro account with 4 seats is extremely concerning.
Hi MrK, sorry for the delay here — is there a reproducible example of the latency that you can share? We're unable to reproduce it our end but that would go a long way in fixing the root cause and making this experience better for you (and everyone else that might run into it)