railway apac metal edge network problem

23 days ago

Crazy roundtrip using metal edge network

my default region is Singapore (all deployment on railway metal) and I already migrate all my deployment to singapore

when I ping my service at https://api.payclusive.web.id/_health (you can try)
the edge network is located in railway/europe-west4 network so it add latency at the other end of the worlds

this make latency insane (+300ms added for roundtrip alone, avg response time is 550ms)
before railway metal+edge network my app response time is around 100ms from Apac to US West Coast back then

thats how bad it is, my deploy using old infrastructure at west coast is faster and responsive than current "near region" edge network is

currently I turn off railway metal edge network and it shave around 100ms (crazy) latency

please someone tell me that this is know issue

Awaiting User Response

28 Replies

23 days ago

update: after turn off railway metal edge network now it pointing to railway/us-west1 (100ms faster) than railway/europe-west4

I trace route and ping via online website and I'am right hetzner vps got 5ms that mean edge network it pointing at europe at the time


20 days ago

Noted, we have a fix in flight to address this on Metal edge.


Status changed to Awaiting User Response railway[bot] 20 days ago


angelo

Noted, we have a fix in flight to address this on Metal edge.

18 days ago

do we have apac metal network or it just pointing dns and route problem


Status changed to Awaiting Railway Response railway[bot] 18 days ago


18 days ago

APAC, but it needs to route correctly via the ISPs which is the root cause of the issues you've seen.


Status changed to Awaiting User Response railway[bot] 18 days ago


16 days ago

Can you please perform a traceroute to 66.33.22.1 from the network that you're experiencing this issue with and share the results with us


14 days ago

sorry late reply
update: few days ago it pointing to southeastasia now it backs to europe west


Status changed to Awaiting Railway Response railway[bot] 14 days ago


phin

Can you please perform a traceroute to 66.33.22.1 from the network that you're experiencing this issue with and share the results with us

14 days ago

wait, it didnt work lol


phin

Can you please perform a traceroute to 66.33.22.1 from the network that you're experiencing this issue with and share the results with us

14 days ago

cant do traceroute or ping


tonyhart7

cant do traceroute or ping

14 days ago

Why not, just curious?


Status changed to Awaiting User Response railway[bot] 14 days ago


angelo

Why not, just curious?

13 days ago

idk

Attachments


Status changed to Awaiting Railway Response railway[bot] 13 days ago


13 days ago

update: now railway edge network is pointing to railway/asia-southeast1-eqsg3a edge and it improve response times but the latency still have a problem

Attachments


13 days ago

this is same project deployed on US West Oregon non metal

Attachments


13 days ago


this is railway metal virginia, the latency certainly better than APAC metal

Attachments


13 days ago

this is california metal, seems like the only problem is APAC metal since latency is comparable to another metal instance

if you need another help for testing let me know

Attachments


tonyhart7

idk

13 days ago

We are asking you to run a trace route on your local computer.


Status changed to Awaiting User Response railway[bot] 13 days ago


brody

We are asking you to run a trace route on your local computer.

13 days ago

if its on local ofcourse its working, I just dont think its my internet problem since I can access other instance normal


Status changed to Awaiting Railway Response railway[bot] 13 days ago


13 days ago

Right but if you are experiencing latency issues we would like you to run the trace route on your local computer.

Put your service and all associated services in the closest region to where you live, use the private network for all service to service communications, and have the Metal Edge enabled, then check the latency, if it's still sub optimal then run the trace route again.


Status changed to Awaiting User Response railway[bot] 13 days ago


brody

Right but if you are experiencing latency issues we would like you to run the trace route on your local computer.Put your service and all associated services in the closest region to where you live, use the private network for all service to service communications, and have the Metal Edge enabled, then check the latency, if it's still sub optimal then run the trace route again.

13 days ago

if its my internet then dashboard latency should not exceed 6000 milisecond, idk why you insist its my internet problem


Status changed to Awaiting Railway Response railway[bot] 13 days ago


13 days ago

also my webserver (rust btw) experiencing timeout to postgress that in the same region and using private network
if you think this is my internet problem idk how I can tell you

again, I mentioning again since the start of the post that this is HAPPEN only when I migrate to APAC metal region not on when I still use US West coast

Attachments


13 days ago

I think there is a misunderstanding here, your original posts mentions latency to your web application, not to the dashboard, please clarify where the problem actually lays.

There are a lot of application level issues that could lead to timeouts when connecting to Postgres, you would want to have more verbose logging so that you can find out the cause of that error.


Status changed to Awaiting User Response railway[bot] 13 days ago


brody

I think there is a misunderstanding here, your original posts mentions latency to your web application, not to the dashboard, please clarify where the problem actually lays.There are a lot of application level issues that could lead to timeouts when connecting to Postgres, you would want to have more verbose logging so that you can find out the cause of that error.

13 days ago

there are multiple problem but they all because the same problem latency
let me articulate what I finding so far in case you didnt read or see photos I upload

1. in my test only railway metal apac that has latency issue
2. you cant blame config or anything since this is the same project that has same configuration only with different region
3. I found it strange when railway apac network edge is pointing to south east asia (singapore) but not having much different latency (200ms,300ms,400ms)
usually when I use GCP,DO singapore server, I got around sub 30ms latency so its very strange for me having a hundred ms latency in singapore server notably because its not making sense since its no better than US west coast deploy

number 3 is what puzzle me since weeks because it says railway/asia-southeast1-eqsg3a in header but having equally if not worse latency with application that deployed on US


Status changed to Awaiting Railway Response railway[bot] 13 days ago


13 days ago

maybe just maybe packet get rerouted or something since my connection to postgress using private network (railway internal) is affected too
also all project is idle with no traffic, I cant imagine with 80% workload


echohackEMPLOYEE

12 days ago

Hey tony,

Hate to sound like a broken record, but we need diagnostics to help continue to fix the issue.

Can you run traceroute 66.33.22.1 from the network you're having this issue with?


Status changed to Awaiting User Response railway[bot] 12 days ago


echohack

Hey tony,Hate to sound like a broken record, but we need diagnostics to help continue to fix the issue.Can you run traceroute 66.33.22.1 from the network you're having this issue with?

10 days ago


is this any help???

Attachments


Status changed to Awaiting Railway Response railway[bot] 10 days ago


8 days ago

Thank you.

Can you please provide numbers for the response times of your health route while using the Railway provided domain, not the Cloudflare domain.

Best,
Brody


Status changed to Awaiting User Response railway[bot] 8 days ago


brody

Thank you.Can you please provide numbers for the response times of your health route while using the Railway provided domain, not the Cloudflare domain.Best,Brody

7 days ago

wait you are right, I got normal latency

now I turn off cloudflare proxy and use dns only, this make latency better even when I use CF
that solve a lot of things but we still need CF to guard our service

but for now its good


Status changed to Awaiting Railway Response railway[bot] 7 days ago


7 days ago

Hello,

Thank you for confirming, we have identified that Cloudflare does not follow the correct routes to the origin server, we have a fix in mind for this, but no ETA yet unfortunately.

Best,
Brody


Status changed to Awaiting User Response railway[bot] 7 days ago


railway[bot]

7 days ago

Hello!

We're acknowledging your issue and attaching a ticket to this thread.

We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.

Please reply to this thread if you have any questions!