From where are the requests being made? And do you only see this latency in logs?

I was seeing the latency in the HTTP logs, but when I turned on caddy logging that went away. However cloudflare still reports that time in the cfOrigin timing

e.g. I have a request id (Ify-vzrQQbabZucVezItjw) where it was 18ms in the app, but Cloudflare reports (and I obsurved) 350ms of latency in Railway

Cloudflare proxied is seeing some increased latency yes

And no, it is not normal, team is aware

YayI I thought I was going nuts 😓

Is it an ongoing incident?

Would disabling the proxy help? 🤔

It would resolve the latency issue yes

Would Railway still handle SSL termination or would I need to enable it in Caddy?

Railway handles that for you

Caddy within Railway is generally unrecommended, unless you specifically need additional behaviour that Railway doesn't support by default

Yeah, I need it to reverse-proxy between 3 different independent backends and potentially other services (marketing SSR, vite PWA and Bun API)

Disabled proxies and response times are down to <30ms! Thank you!!!

WIll let you know once the proxied issue is resolved

Thank you!

That annoying cloudflare trakcing script isn't included anymore either. Much better ❤️

Thank you so so much, I would never have guessed that

Ohh, this is really interesting... US, EU and Asia monitors all dropped but Australia specifically shot up from having the best times to the worst (~500ms). We don't do buisness there but that is a facinating find 🤔 . I wonder if cloudflare has a way around their one cable bottleneck that only applies to the proxy

Could you run a traceroute to 69.46.46.46 from Australia?

No, better stack controls those servers not me 🙁

Its just a regional heartbeat monitor

We have an Australia CDN POP in the works - once that’s online, you’ll see better latency from AUS

Update: the Melbourne POP is online now 🙂

Nice! I have no buisness need in AUS but I found it to be a very interesting side effect and worth sharing 🙂

We've been seeing this consistently, but I am not sure where the issue lies exactly.

Railway Metrics look good on the container, though they are very generalized (p50 etc), so I'm not sure how much I trust them.

For requests in the browser, we see this time mostly under "Waiting for server response". We've profiled our application, and there is a significant mismatch. For some requests, we handle these in <2ms, but both Railway and the browser report in excess of 150ms.

I thought this was maybe an nginx/fpm issue, but if I SSH into the container and execute requests locally with curl, we see figures much closer to what we've profiled on the PHP side.

FWIW, we've disabled Cloudflare Proxy for these tests (it's been off for days, actually).

I get the feeling that the issue is somewhere between Railway Edge <-> RW Proxy, or between

RW Proxy <-> Container, but I'm not sure what I can do to identify which.

Just an example (request IDs match):

Please could you answer the following so I can understand if this is expected latency or a routing issue:

• Where you are in the world (country)
• Where your Railway deployment is located (region)
• The value of pop when you visit https://railway.com/.railway/cdn-trace

I did create my own thread (https://discord.com/channels/713503345364697088/1514099639710912563) for this, if you'd prefer to keep it all there? I'm not sure it's the same issue.

Let’s talk there then

@Thaumanovic according to the bot at least upstreamRqDuration is the reported latency from the edge to the application... in my case it was also reporting long times when using the cloudflare proxy.

Not sure if that is helpful. Best of luck ❤️