I'm in Edinburgh, Scotland (UK). https://railway.com/.railway/cdn-trace shows:

v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dpber-1
pop=ber1
node=bpxb
ts=1781067039462
hint=Railway is coming to a city near you

Hi @Thaumanovic, could you provide the information I asked from you in the previous thread? 🙂

See above. 🙂 Deployment is us-east

I have a user in Newark, close to Ashburn. They report:

v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dpjfk-1
pop=jfk1
node=57w5
ts=1781067324746
hint=Railway is coming to a city near you

They also experience this latency.

Are you also experiencing the latency personally?

Yes, all of our users appear to be

Approximately when did this start? I’m expecting this is an application issue, but it could be a side effect of new HTTP/2 behavior on our edge.

It's hard to say. Our app is old, so we expected signficant performance impact when we migrated ~3-4 weeks ago. We've done a heavy optimization pass since then, and our own profiling shows good improvement, but those gains don't seem to be realized because of this issue (wherever it may be).

Understood, well we only moved applications to our new edge starting mid-last week, so I don’t think that’s related. Unfortunately I believe this is an application issue. upstreamReqDuration is the time it takes for your application to answer our origin proxies.

That's what I understand it to be, but these figures:

(upstreamRqDuration | Browser TTFB)
2ms | 205ms
2ms | 356ms

Are rather far apart.

Obviously I'd expect some latency from distance and overheads, but I would not expect that much, nor would I expect it to be as inconsistent as it is.

I have profiled that endpoint, and it shows a near-consistent ~1.5ms request time (it just renders a custom SVG). The fact that upstreamRqDuration is similar rules out that it's an nginx<->fpm issue (we'd maybe see that under heavy load), so I am not convinced it's an application issue. 🙁

Do you also experience high latency with this endpoint? For the user in Orlando, it’d be ideal to get the Railway CDN trace information from them.

Let me hit it just now for a fresh figure

(OL user is showing away, probably sleep now - but I can follow up with that)

CHg5SuUHSQKgPMGzg4a9AQ

(Not sure the req ID is any use to you, but there is is anyways)

That was prod, so it has CF proxy, but even on dev the figures are only marginally smaller.

Does upstreamRqDuration measure a complete response, or is it similar to TTFB?

If the latter, then the issue could lie in our setup (requests not terminating properly, maybe?), but if the former then I am certain it's not our application.

We do use gzip, so it's chunked transfers without a Content-Length header, but this is quite a common thing so I'd be surprised if that were somehow confusing the proxy.

Where are you based? (Country)

Scotland

Which pop do you hit on https://railway.com/.railway/cdn-trace ?

pop=ber1

node=mmj8

Hmm Berlin, ideally you should hit the London pop. Could you provide the name of your ISP and a traceroute to 69.46.46.1 if possible?

> Your ISP is

> Vodafone Limited

 tracert 69.46.46.1

Tracing route to 69.46.46.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     9 ms     8 ms     8 ms  90.247.128.1
  3    19 ms    18 ms    18 ms  63.130.172.45
  4    20 ms    20 ms    20 ms  ae1-0.lon10.core-backbone.com [195.66.224.238]
  5    37 ms    37 ms    37 ms  ae5-2089.ber10.core-backbone.com [81.95.9.229]
  6    41 ms    41 ms    41 ms  coreb-ber.cdn77.com [138.199.1.12]
  7    42 ms    41 ms    41 ms  vl221.ber-ipb-dist-1.cdn77.com [79.127.195.227]
  8    41 ms    41 ms    41 ms  69.46.46.1

Cool thank you, I’ll look into fixing this routing today. When did you last deploy the nginx server btw? We made some further improvements to internal routing earlier this week but it requires a redeploy.

10 hours ago

May I also ask why you’re using nginx on top of your application?

PHP usually requires something in front of it (nginx, apache, etc) to dish out requests. It's all a single container: https://github.com/shinsenter/php (nginx-fpm flavour)

Ideally we'd go for something like RoadRunner, but our app architecture does not support that at this time.

Would it be possible to get the URL to the SVG you’re talking about that has high latency so I can verify end to end?

Err, 1 sec. Might be auth-walled, let me check.

Is it not. Can I DM the URL to you?

Of course 🙂

Hi, could you check https://railway.com/.railway/cdn-trace again?

pop=cdg1

node=e9jw

So it's shifted, just not to the right place 🥲

cdg = france?

Could you traceroute again?

Yeah

interesting, not sure why vodafone's doing that.. our London POP has a PNI (interconnect) directly with their network in London

Well

The answer to that is probably that it's Vodafone

I'd be on an 8 gig connection with YouFibre if I could, but surprisingly for a building built in the last 10 years, we have no fibre. Good old ADSL.

Do you speak to someone at Vodafone, or is it the CDN provider, or can I harass their engineers given I am unable to subscribe from their marketting SMS and emails?

Our Orlando user sees pop=atl1 (Verizon)

Not sure what's up with this one, but hitting my service via CF still gives me the Berlin POP - directly is CDG.

Just adding a note here - I am based in Seattle, and my CF traffic seems to route through Berlin as well. (direct is sjc1)

quMwwRdrQOSXz9KNU79b0g | ingress=ber1 (Berlin) | ttfb=341ms

By2JgMnuRjiNRzALBhdwDg | ingress=ber1 (Berlin) | ttfb=340ms

otdQeHxTRe-AXN0QyCLmYg | ingress=ber1 (Berlin) | ttfb=341ms

Should be fixed

I contacted Vodafone but can't make any promises. I'll let you know if they get it resolved.

CDG isn't too far from London in the grand scheme of things. The latency is still great.

I have a similar problem on my domain behind a cloudflare proxy.

I'm in Paris, the deployment is in eu-west and I have between 400 and 900ms

May I ask what you use the Cloudflare proxy for? It'd be ideal for us if users could disable the CF proxy, as they keep messing with routing to our edge.

Other than CDN, bot prevention and redirect rules in our case.

Hmm ok, we have CDN + bot prevention now. We'll work on "page-rules"-esque features soon.

CDN:

WAF/bot protection:

Analytics, DDoS protection and custom security rules (even if the server is not "under attack", some bot or ASN can have a real impact on the billing)

We can maybe move this to our application itself to be honest. Was just more convenient to punt it to Cloudflare so we didn't have to handle the hits.

We did it with 301 redirects so I'm hoping all of the crawlers and stuff will stop hitting the old endpoints

Could you tell me whether someone is already looking into this issue, or if I should open a new thread?

To be honest, I’m not sure whether this is a Railway issue or a Cloudflare issue. A few months ago, similar routing problems showed an X-Railway-Edge header from another region, but this time:

• the region seems correct (europe-west4)
• I’m getting an average response time of around 500 ms from Paris
• Railway’s metrics show an unexpectedly low response time (p90 used to be ~600 ms and is now ~30 ms)

The issue is that Cloudflare in Paris is routing to our POP in Tokyo.

https://cf-trace.up.railway.app/cdg/69.46.46.66

It's not necessarily a Railway issue, as we do not operate the Cloudflare network or proxy and would generally advise customers to turn it off to resolve the issue.

We still of course would like to support Cloudflare's network, so we will work to resolve this issue, however they have increasingly become harder to support as of recent.

Alright, could you clarify what you mean by “we will work on it”?

Is the issue currently being addressed, or should I open a ticket on Cloudflare’s side?

Let me know if there’s anything I can do on my end to help speed up the fix.

What about cases where are not running CF Proxy? Our dev environment has this disabled (tho still uses CF DNS) and we still see a large mismatch between our actual response times and TTFB.

I would need more information. Which POP are you being routed to? If it's a web URL, what is the breakdown to TTFB in your browser DevTools? What is the discrepancy between the two?

This would be helpful.

Can sort that out just now, gimmie 10

I generally try to fix these routes every couple of days, but they keep breaking. So I can't make any promises. If you could make a ticket with Cloudflare referencing the traceroute, that would be ideal and might escalate it on their side too.

That being said, I think it should be fixed in most of their colos now.

Yes, it seems to be fixed

Thank you @Phineas and good luck with these bad boys, I hope it'll get better

No worries and thank you!

Traceroute:

Tracing route to nejrnq6f.up.railway.app [69.46.46.60]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.0.1
  2     8 ms     9 ms     8 ms  90.247.128.1
  3     *        *        *     Request timed out.
  4    17 ms    15 ms    16 ms  ae5-100-xcr1.man.cw.net [195.89.96.113]
  5    21 ms    20 ms    20 ms  ae31-xcr1.lns.cw.net [195.2.9.97]
  6     *        *        *     Request timed out.
  7    27 ms    28 ms    28 ms  193.251.128.71
  8    27 ms    27 ms    27 ms  81.52.186.230
  9    28 ms    27 ms    27 ms  vl221.par-itx5-dist-2.cdn77.com [79.127.195.36]
 10    27 ms    27 ms    27 ms  69.46.46.60

Browser timing:

Real request time in our app:

Thanks. Could I see the response headers?

Sorry for the formatting, DevTools butchers it

Thanks, and sorry which Railway service was this for? (You can link me to it on the dashboard)

https://railway.com/project/0cddb90a-9f25-46a3-84ed-0be23eca109d/service/e6418310-bfdf-444d-822c-8d07d547cbec

(that do?)

Environment ID: e70b963a-417f-4e0b-8289-cdd95800254c

Could you visit https://us-east4-eqdc4a-production.up.railway.app/ (refresh twice) and then show me the Timing stats?

Timing from that last one:

ReqId: fyWowW6HTxyHG7pj8u2xcg

Seeing: cdg1.8vsn

Thanks. That seems generally correct for your geography (considering the cdg reroute)

The extra 100ms you see is likely coming from your application. What is the actual response from that route?

On our app?

Yeah, what does this (loader.php) actually respond with - what's the content?

Just a bunch of HTML, that particular one was 1.6kB

the issue is back

Yeah I brought the issue back so that their team can debug, apologies. https://x.com/phineyes/status/2065546050719604848?s=46

Thanks for the update - appreciate all your efforts on this! ❤️

The issue is affecting us as well, our servers are located in singapore

@Thaumanovic Apparently they've made some changes, could you check again?

At my day job just now, can check in a few hours tho

Got my roomie to check:

v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dpams-1
pop=ams1
node=kxr8
ts=1781540171037
hint=Railway is coming to a city near you