2 months ago
app is hosted in us-east, while i'm in new zealand. this should be a ~250ms round trip, but whenever i enable the CF proxy on the domain, latency spikes.
image 1 is when using a domain that bypasses cloudflare, or turning the CF proxy off
image 2 is when the CF proxy is enabled
image 3 is a friend in us-west bypassing the CF proxy
image 4 is the same friend accessing via the CF proxy
additional context:
when i refer to image 1, this is both: using the up.railway.app domain, or my custom domain with proxying turned off in cloudflare
due to the issue reproducing when disabling the proxy on the same domain, i'm almost convinced the issue is not in code, as the deployment doesn't change when simply disabling the cloudflare proxy. the only thing i haven't isolated is the possibility of cloudflare not handling
no-transformcorrectly, as my bun web server does compression at origin and sets theno-transformcache control header to prevent an additional compression stepi have also upgraded to cloudflare pro for this domain with no improvement
as per a suggestion in the cloudflare discord, there's no issues at the CF colo i get connected to
i did reply to mahmoud on twitter about this using a brand new deployment of a blank nodejs/astro template as it seems to reproduce easily, so again i don't believe it's an issue with my code or runtime: https://x.com/rfxkairu/status/2007082744824934488
this is with argo enabled too, disabling it makes the problem even worse.
i've posted about this before: https://station.railway.com/questions/high-http-response-times-w-poor-routing-0062c88d
but "just move it closer" isn't really a fix as this clearly isn't a distance issue, i'm not trying to beat the laws of physics here. i've exhausted just about every option i can think of.
this is the last thing that's blocking me from getting my railway migration going live, and i mostly just want to know where the issue lies. if it's on me, railway or cloudflare. at the moment my only options are to go without proxying and incur heavier bandwidth costs, keep proxying and incur a worse user experience, or i move the web app off railway to somewhere else. neither of which are particularly good options.
happy to run diagnostics, tweak settings, make code changes to debug or whatever to get this figured out.
project id: bc0dbee7-3f1f-41f9-b5b1-b639fa883ca9
service id: 03f1567d-aea5-4d9b-898d-efcacc5e96cb
environment id: 7ff93a1b-6c57-4271-a659-4b8880fb59a7
18 Replies
i've managed to isolate a large chunk of the latency down to a cache rule. disabling that brings latency through cloudflare to around 550ms, which i believe is down to poor routing when hitting the railway edge like i mentioned in the previous post. still a far cry from 250ms but much better than 1500ms.
when i connect to my railway.app domain, the edge i hit is us-west
when i disable the cf proxy, i hit us-west
when i enable the cf proxy, i hit asia-southeast1
this doesn't make any sense when NZ has two cloudflare PoP's. at no point should NZ traffic be routed to singapore to get to us-east. who do i need to bother to get this addressed? railway or cloudflare?
if it helps, i did notice that my deployment routes me to the AKL PoP, while accessing railway.app routes me to CHC. geographically i'm closer to CHC so i'm not sure what's going on there
2 months ago
We are encountering the exact same problem.
2 months ago
Hi dude, I'm having the same problem, did you manage to figure this out?
cache rule still blows the latency up but the routing issue appears to be fixed https://discord.com/channels/713503345364697088/1461175819991908404/1461409441395249276
2 months ago
Same here, without Cloudflare proxy I hit railway/europe-west4 (the correct one, 60ms), when I activate Cloudflare proxy i hit railway/us-east4 (700 ms), the railway server is located in europe-west4
2 months ago
Isn't New Zealand geographically closest to Singapore? at least in comparison to our PoP in US-West.
we have a direct connection to us west at about 140ish ms which is why routing through singapore makes no sense
2 months ago
Our edge network is an anycast network, so being routed to our closest PoP makes sense, I am not sure what the issue is here.
if it were just an additional ~230ms i can live with that, but it's not. first image is bypassing cloudflare (which goes to us-west edge), the other is through cloudflare which goes to asia-southeast
railway correctly routes me to US PoP's, it's when the traffic comes from cloudflare it gets it wrong
2 months ago
You mentioned you are in new Zealand, so it's the other way around, routing is wrong when not using Cloudflare
the fix yesterday was solid, going through cf was around 400-600ms https://discord.com/channels/713503345364697088/1461175819991908404/1461428714612064463
2 months ago
The optimal routing is by distance.
i get that this isn't the most scientific since it's just wonder network but it illustrates my point
yes NZ is geographically closer to SG but NZ to NY is the same latency as NZ to SG. whatever change was made yesterday was simply faster