a month ago
I am seeing an average of 2-400ms of latency on my service however pinging into my Caddy gateway and pinging localhost with curl has ~3ms response times.
All services are deployed to the US East region and being pinged from NYC and many PoPs from betterstack where 200ms has been found to be the floor.
Is there anything we can do to mitigate this as that is quite a high floor for every single API request, page load, navigation...
7 Replies
a month ago
Your HTTP request logs include totalDuration and upstreamRqDuration fields. If totalDuration is high but upstreamRqDuration is low, the latency is in the network/edge path rather than your application. You can filter for slow requests with @totalDuration:>200 and compare both values to pinpoint where the time is spent, which will help us narrow this down.
Status changed to Awaiting User Response Railway • 28 days ago
Railway
Your HTTP request logs include `totalDuration` and `upstreamRqDuration` fields. If `totalDuration` is high but `upstreamRqDuration` is low, the latency is in the network/edge path rather than your application. You can filter for slow requests with `@totalDuration:>200` and compare both values to pinpoint where the time is spent, which will help us narrow this down.
a month ago
Where can I find these logs? I have done extensive testing by pinging from the container on the gateway. This 2-400ms floor makes the page load time ~2-3 seconds due to CSS and JS files also being impacted
Status changed to Awaiting Railway Response Railway • 28 days ago
a month ago
You can find the HTTP request logs by clicking on a deployment within your service and switching to the "HTTP Logs" tab. From there, filter with @totalDuration:>200 and compare totalDuration (full round-trip including edge routing) against upstreamRqDuration (your app's response time) to pinpoint whether the latency is in the network/edge path or your application.
Status changed to Awaiting User Response Railway • 28 days ago
Railway
Your HTTP request logs include `totalDuration` and `upstreamRqDuration` fields. If `totalDuration` is high but `upstreamRqDuration` is low, the latency is in the network/edge path rather than your application. You can filter for slow requests with `@totalDuration:>200` and compare both values to pinpoint where the time is spent, which will help us narrow this down.
a month ago
upstreamRqDuration matches totalDuration and both are high, but when signing into the container these numbers do not hold up.
Status changed to Awaiting Railway Response Railway • 28 days ago
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 28 days ago
a month ago
Hey bot. Can we escalate this for someone to see at least. 2s response times on a static site is really really bad.
I enabled logs in Caddy and it is saying 5-20ms response times in the application layer...
a month ago
I just recorded a great example. I hope the reqeust info is not PII
When searching the services logs
@request.headers.X-Railway-Request-Id[0]:Ify-vzrQQbabZucVezItjw
It returns "duration": 0.015677358,
In the HTTP logs it says 17/22ms
However Cloudflare recorded 350ms of latency from Railway
I sadly cannot record the full network trace but it is provided on Discord:
https://discord.com/channels/713503345364697088/1513708831941726318/1513711026166566952
Attachments
a month ago
There is apparently a known latency issue in US-East with Cloudflare proxy. Disabling that worked. Thanks Bot and Astrid!
Follow the discord thread for updates if you stumble upon this in the future
Status changed to Open 0x5b62656e5d • 28 days ago
Status changed to Solved 0x5b62656e5d • 28 days ago