14 days ago
Something is going on with the network connections from Railway. Noticed some users complaining, and after some tests, users from LATAM are getting response times of 22 - 30 seconds, while users outside of LATAM still get the usual 0.01 - 0.1 seconds.
What is going on? Such a huge lag is causing timeouts on all users.
78 Replies
Postman test from Argentina
Attachments
Requests on the server from online players outside of latam
Attachments
14 days ago
Thanks for the update 🙏
Don't know what's going on, but I changed nothing on the server or anything
14 days ago
I'm figuring it out right now, sorry for the delay
14 days ago
thanks for the updates
14 days ago
I've informed the team
14 days ago
Kuha, where geographically are most of your users from? Or are they global?
14 days ago
So South America generally?
14 days ago
Thank you 🙏 that helps a lot
14 days ago
I appreciate it, I'll let you know, thank you 🙂
14 days ago
Thank you for the info <:Pray:1360710432029147186>
14 days ago
The same thing happens to me, Peru!
14 days ago
It'd be helpful if someone experiencing slow requests could run an mtr or traceroute to 69.46.46.46
Attachments
14 days ago
Hmm could you install the mtr tool and try that instead?
14 days ago
ping (15 pkts): min/avg/max/stddev = 69.4 / 92.9 / 164.2 / 28.9 ms, 0% packet loss (but notable jitter, σ≈29 ms).
traceroute reaches Miami cleanly and dies at CDN77 (ICMP filtered past it):
3 telefonicaglobalsolutions.com ~9 ms
7-9 *.mia03.atlas.cogentco.com (Miami) ~70-143 ms
10 vl221.mia-eq6-dist-2.cdn77.com ~70 ms
11-30 * * * (filtered, but 69.46.46.46 answers ICMP, ttl=54)
Key signal: public network from LATAM to your Miami edge is healthy (~93 ms, 0% loss), but HTTP TTFB to my service is 2–18 s (intermittent). My app's upstreamRqDuration is ~166 ms; the time is lost in the
edge→origin path. My service is deployed in us-west2 — looks like the edge(Miami)→us-west2 backhaul or the proxy is the bottleneck, not the LATAM→Miami public path. Service: web / project insightful-courage.
Slow request-IDs: 9l_uuPl7RyOvkhiKlt7tkg, 7aZv4ptZTZ2R2J4wWUN5dQ.
this is from Argentina
Attachments
@Phineas
Attachments
14 days ago
Thank you
14 days ago
We're investigating
Attachments
14 days ago
We have taken the Miami POP offline which was experiencing elevated packet loss.
14 days ago
It should be fixed now, and we will bring it back online once we the issue is resolved.
14 days ago
Im having the same issue. It's just too slow
14 days ago
Update from Peru, isolating the remaining latency:
- Client → edge atl1 (69.46.46.123): clean — ping 0% loss, avg 102 ms (RTT to Atlanta, expected).
- App processing: healthy — gunicorn upstream time avg 86 ms (125 reqs, max 1.3 s), origin in us-west2.
- But external TTFB is 2–3 s, spiking to 22 s.
So with Miami dropped, a clean client→atl1 path, and a fast origin, the ~2+ seconds unaccounted for are in the atl1→us-west2 edge↔origin backhaul — that's where the bottleneck moved. The Atlanta→us-west2 internal path looks congested/lossy. Happy to run more tests if useful.
14 days ago
The issue is with a backbone internet service provider, Arelion. We use them in US West so lots of internet traffic into US West is seeing issues. We're working on mitigating this.
14 days ago
Test made by my side
try1: ttfb=13.147113s total=13.147166s
try2: ttfb=7.557014s total=7.557089s
try3: ttfb=6.340120s total=6.340174s
try4: ttfb=15.468342s total=15.468453s
try5: ttfb=12.789899s total=12.789947s
try6: ttfb=1.077543s total=1.077596s
14 days ago
we've dropped Arelion AS1299 from our US-West region, you should start to see improvement shortly
6 days ago
i let the team know
6 days ago
same here (Chile)
6 days ago
Team is working on it now
6 days ago
I have no idea, I believe the team is investigating the cause right now
6 days ago
will report back if I get any more info
6 days ago
Are you opening an incident?
6 days ago
Incident just called:
6 days ago
I haven't been told whether it even is the same issue as last time, although if it is then yes it could be redirected
Our clients are in LATAM and they are all reporting the problem. We're in emergency mode.