23 days ago
Crazy roundtrip using metal edge network
my default region is Singapore (all deployment on railway metal) and I already migrate all my deployment to singapore
when I ping my service at https://api.payclusive.web.id/_health (you can try)
the edge network is located in railway/europe-west4 network so it add latency at the other end of the worlds
this make latency insane (+300ms added for roundtrip alone, avg response time is 550ms)
before railway metal+edge network my app response time is around 100ms from Apac to US West Coast back then
thats how bad it is, my deploy using old infrastructure at west coast is faster and responsive than current "near region" edge network is
currently I turn off railway metal edge network and it shave around 100ms (crazy) latency
please someone tell me that this is know issue
28 Replies
23 days ago
update: after turn off railway metal edge network now it pointing to railway/us-west1 (100ms faster) than railway/europe-west4
I trace route and ping via online website and I'am right hetzner vps got 5ms that mean edge network it pointing at europe at the time
Status changed to Awaiting User Response railway[bot] • 20 days ago
angelo
Noted, we have a fix in flight to address this on Metal edge.
18 days ago
do we have apac metal network or it just pointing dns and route problem
Status changed to Awaiting Railway Response railway[bot] • 18 days ago
18 days ago
APAC, but it needs to route correctly via the ISPs which is the root cause of the issues you've seen.
Status changed to Awaiting User Response railway[bot] • 18 days ago
16 days ago
Can you please perform a traceroute to 66.33.22.1
from the network that you're experiencing this issue with and share the results with us
14 days ago
sorry late reply
update: few days ago it pointing to southeastasia now it backs to europe west
Status changed to Awaiting Railway Response railway[bot] • 14 days ago
phin
Can you please perform a traceroute to 66.33.22.1 from the network that you're experiencing this issue with and share the results with us
14 days ago
wait, it didnt work lol
phin
Can you please perform a traceroute to 66.33.22.1 from the network that you're experiencing this issue with and share the results with us
14 days ago
cant do traceroute or ping
Status changed to Awaiting User Response railway[bot] • 14 days ago
Status changed to Awaiting Railway Response railway[bot] • 13 days ago
13 days ago
update: now railway edge network is pointing to railway/asia-southeast1-eqsg3a edge and it improve response times but the latency still have a problem
Attachments
13 days ago
this is railway metal virginia, the latency certainly better than APAC metal
Attachments
13 days ago
this is california metal, seems like the only problem is APAC metal since latency is comparable to another metal instance
if you need another help for testing let me know
Attachments
Status changed to Awaiting User Response railway[bot] • 13 days ago
brody
We are asking you to run a trace route on your local computer.
13 days ago
if its on local ofcourse its working, I just dont think its my internet problem since I can access other instance normal
Status changed to Awaiting Railway Response railway[bot] • 13 days ago
13 days ago
Right but if you are experiencing latency issues we would like you to run the trace route on your local computer.
Put your service and all associated services in the closest region to where you live, use the private network for all service to service communications, and have the Metal Edge enabled, then check the latency, if it's still sub optimal then run the trace route again.
Status changed to Awaiting User Response railway[bot] • 13 days ago
brody
Right but if you are experiencing latency issues we would like you to run the trace route on your local computer.Put your service and all associated services in the closest region to where you live, use the private network for all service to service communications, and have the Metal Edge enabled, then check the latency, if it's still sub optimal then run the trace route again.
13 days ago
if its my internet then dashboard latency should not exceed 6000 milisecond, idk why you insist its my internet problem
Status changed to Awaiting Railway Response railway[bot] • 13 days ago
13 days ago
also my webserver (rust btw) experiencing timeout to postgress that in the same region and using private network
if you think this is my internet problem idk how I can tell you
again, I mentioning again since the start of the post that this is HAPPEN only when I migrate to APAC metal region not on when I still use US West coast
Attachments
13 days ago
I think there is a misunderstanding here, your original posts mentions latency to your web application, not to the dashboard, please clarify where the problem actually lays.
There are a lot of application level issues that could lead to timeouts when connecting to Postgres, you would want to have more verbose logging so that you can find out the cause of that error.
Status changed to Awaiting User Response railway[bot] • 13 days ago
brody
I think there is a misunderstanding here, your original posts mentions latency to your web application, not to the dashboard, please clarify where the problem actually lays.There are a lot of application level issues that could lead to timeouts when connecting to Postgres, you would want to have more verbose logging so that you can find out the cause of that error.
13 days ago
there are multiple problem but they all because the same problem latency
let me articulate what I finding so far in case you didnt read or see photos I upload
1. in my test only railway metal apac that has latency issue
2. you cant blame config or anything since this is the same project that has same configuration only with different region
3. I found it strange when railway apac network edge is pointing to south east asia (singapore) but not having much different latency (200ms,300ms,400ms)
usually when I use GCP,DO singapore server, I got around sub 30ms latency so its very strange for me having a hundred ms latency in singapore server notably because its not making sense since its no better than US west coast deploy
number 3 is what puzzle me since weeks because it says railway/asia-southeast1-eqsg3a in header but having equally if not worse latency with application that deployed on US
Status changed to Awaiting Railway Response railway[bot] • 13 days ago
13 days ago
maybe just maybe packet get rerouted or something since my connection to postgress using private network (railway internal) is affected too
also all project is idle with no traffic, I cant imagine with 80% workload
12 days ago
Hey tony,
Hate to sound like a broken record, but we need diagnostics to help continue to fix the issue.
Can you run traceroute 66.33.22.1
from the network you're having this issue with?
Status changed to Awaiting User Response railway[bot] • 12 days ago
echohack
Hey tony,Hate to sound like a broken record, but we need diagnostics to help continue to fix the issue.Can you run traceroute 66.33.22.1 from the network you're having this issue with?
10 days ago
is this any help???
Attachments
Status changed to Awaiting Railway Response railway[bot] • 10 days ago
8 days ago
Thank you.
Can you please provide numbers for the response times of your health route while using the Railway provided domain, not the Cloudflare domain.
Best,
Brody
Status changed to Awaiting User Response railway[bot] • 8 days ago
brody
Thank you.Can you please provide numbers for the response times of your health route while using the Railway provided domain, not the Cloudflare domain.Best,Brody
7 days ago
wait you are right, I got normal latency
now I turn off cloudflare proxy and use dns only, this make latency better even when I use CF
that solve a lot of things but we still need CF to guard our service
but for now its good
Status changed to Awaiting Railway Response railway[bot] • 7 days ago
7 days ago
Hello,
Thank you for confirming, we have identified that Cloudflare does not follow the correct routes to the origin server, we have a fix in mind for this, but no ETA yet unfortunately.
Best,
Brody
Status changed to Awaiting User Response railway[bot] • 7 days ago
railway[bot]
7 days ago
Hello!
We're acknowledging your issue and attaching a ticket to this thread.
We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.
Please reply to this thread if you have any questions!