21 days ago
Hey there, I'm seeing my endpoints take 2-4x the time. I'm still in the early phase of my project without any users yet, but I wanted to shout it out in case there is an issue.
61 Replies
21 days ago
Hello! May I ask whether your app is proxied by Cloudflare?
Btw, I can confirm my service is doing well. Here are my slowest endpoints:
Attachments
21 days ago
Please could you go to https:///.railway/cdn-trace and show me the output? 🙂
v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dplhr-2
pop=lhr1
node=1t7d
ts=1780512397432
hint=Railway is coming to a city near youMy /me endpoint used to take 10-30ms roundtrip and now it is taking 250-350ms and is sometimes stalling up to a few seconds.
21 days ago
Is this anecdotal or from monitoring?
21 days ago
Are you based around or near London? If not, then you are being incorrectly routed to our edge.
21 days ago
Please could you run a traceroute or MTR (preferably) to 69.46.46.46?
This is anecdotal, but I'm seeing the same behavior in different devices. I'm based in Chicago
21 days ago
I am based near Chicago and I am having similar issues.
when I switch over to mobile data on my phone it is fast again, so could it be an issue with my wifi? It wasn't like this before.
21 days ago
Not with wifi specifically, but with your network/ISP. If you send me a traceroute, I’ll be able to engineer the Internet traffic via BGP so that you hit the correct POP.
21 days ago
I know we have issues with Google Fiber at the moment - is that your ISP?
No that's actually my mobile data provider which things are working well with.
I sent the traceroute above btw and also sent mtr results via DMs
Edit: Sorry meant Google Fi, they're different so nvm.
21 days ago
Thank you. Will work on this soon get back to you within the hour or so.
okay tysm! It's not urgent for me, but it might be a broader issue worth looking into. I'll check in again after about an hour!
21 days ago
We’re currently shifting traffic onto our new edge network which contains different peering arrangements / upstream ISPs worldwide, so we’re still in the process of working out some teething issues with certain ISPs!
21 days ago
Experiencing the same issue
21 days ago
Ok, so this is actually a bit harder to engineer/fix. For some reason, your ISP Astound AS6079 has a PNI (network interconnect) with Liberty Global in Chicago. Liberty Global is primarily a European based transit network, and we only announce routes to them in Europe. The issue is, Liberty Global is a pretty bad network and doesn't let us apply traffic engineering policies to routes we announce to them - but we need to use them to reach major ISPs in Europe.
I'm slightly confused why Astound directly upstreams to Liberty Global - it's only useful for reaching European eyeball networks. I'm going to have to contact them directly to fix this.
Sorry for the wall of text - just trying to give context on why I can't fix this immediately.
21 days ago
We're experiencing extreme latency for at least the last hour and a half from all users on many different ISPs hundreds and/or thousands of miles apart.
21 days ago
v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dplax-2
pop=lax1
node=v9kt
ts=1780517402605
hint=Railway is coming to a city near you
21 days ago
from another network:
v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dplax-1
pop=lax1
node=ez9k
ts=1780517322918
hint=Railway is coming to a city near you
21 days ago
Any news?
21 days ago
We're experiencing significant latency. It is intermittent, but has been happening for ~2 hrs now.
21 days ago
Actively investigating. Can you give me a rundown of specifics, esp if you have APM, charts ideal
21 days ago
We are experiencing the same thing. Is this any update or is there at least any update on your status page?
21 days ago
Seeing the same issue intermittently
21 days ago
I just opened up a separate thread. Same issue.
21 days ago
why is this not on the status page?
willscottrod
21 days ago
having the same issue
willscottrod
21 days ago
This is what ours looks like too, and over the last 30 days as well.
It looks like a ton of other people are having this issue and noticing a slowness in request time. I definitely think this should be put on the status page and let customers know what there's a degradation.
21 days ago
Hey folks, we're investigating this right now and will update the status page very soon. Apologies for the wait!
21 days ago
We've been seeing issues as well, since May 27th:
Attachments
21 days ago
It has become worse in recent days - we're on US East and we're seeing it impact users globally.
21 days ago
we've been experiencing this issue on our response time for over a month now, but today is the first time it has become noticeable
21 days ago
cc @SeattleDan
I'm on US East en seeing latency issues, even admin or login post taking very long .
21 days ago
Hey all, we're actively investigating the issue. We just rolled out some changes. Could you please retry and let us know if you are seeing any improvements?
21 days ago
No improvements on my end.
chandrika
Hey all, we're actively investigating the issue. We just rolled out some changes. Could you please retry and let us know if you are seeing any improvements?
21 days ago
No improvements on US east., there is latency degradation also on US east!.
21 days ago
Seems to be some improvement here. I'll keep monitoring.
Attachments
21 days ago
Thank you for all the detail so far folks. If you're still seeing slowness, two quick things would let us confirm you're on the affected path:
NOTE: this is a public thread, please OMIT your domain if you do not want to share it publicly
- Open https://YOUR-APP-DOMAIN/.railway/cdn-trace in your browser and paste just the output here (it shows which edge you're routed through, e.g. pop=lax1)
- Your rough location (city or region is plenty).
Optional but useful: a traceroute/MTR, or a timed request so we can see where the delay is:
curl -w "TTFB:%{time_starttransfer}s total:%{time_total}s\n" -o /dev/null -s https://YOUR-APP-DOMAIN/
21 days ago
That URL just hits my own service, resulting in a 403.
(We do not have CDN enabled - I suspect that's why.)
chandrika
Thank you for all the detail so far folks. If you're still seeing slowness, two quick things would let us confirm you're on the affected path: **NOTE: this is a public thread, please OMIT your domain if you do not want to share it publicly** 1. Open https://YOUR-APP-DOMAIN/.railway/cdn-trace in your browser and paste just the output here (it shows which edge you're routed through, e.g. pop=lax1) 2. Your rough location (city or region is plenty). Optional but useful: a traceroute/MTR, or a timed request so we can see where the delay is: `curl -w "TTFB:%{time_starttransfer}s total:%{time_total}s\n" -o /dev/null -s https://YOUR-APP-DOMAIN/`
21 days ago
v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dpatl-2
pop=atl1
node=1v22
ts=1780531432367
hint=Railway is coming to a city near you
v=hikari:0.1.2:1305a1583dc517b7e2603638b5710b21
r=production-hikari-dpatl-1
pop=atl1
node=mwdw
ts=1780531451203
hint=Railway is coming to a city near you
19 days ago
Hi @amanuel, unfortunately Astound has not yet responded - but we have depeered with Liberty Global, globally. Please let me know if you're hitting a closer POP now 🙂
18 days ago
We've seen improvement on US East since June 4th 17:00 UTC, so whatever changed around then was the likely the culprit for us.
Attachments