Hey there! We've found the following might help you get unblocked faster:

• 🧵 Very high and inconsistent round trip times to backend from end user

• 📚 Deploy Static Sites with Zero Configuration and Custom Domains

• 🧵 502 Bad Gateway error despite successful container startup

If you find the answer from one of these, please let us know by solving the thread!

Hey,

This sounds like a routing issue. Can you give us some more diagnostic information and run a few commands for me?

• The x-railway-edge header from your http request

• traceroute to 66.33.22.11

• traceroute to 66.33.22.1

• The deployment ID that is experiencing the issue

I also have the requestId from your screenshot which should help.

Sharing the request HAR file along with traceroute to 66.33.22.11 and 66.33.22.1

The host is in BLR. Not sure why the edge is railway/europe-west4-drams3a

traceroute to 66.33.22.11 (66.33.22.11), 64 hops max, 40 byte packets

1 192.168.1.1 (192.168.1.1) 3.738 ms 4.312 ms 3.940 ms

2 10.240.9.208 (10.240.9.208) 5.966 ms 5.854 ms 11.043 ms

3 * * *

4 125.18.215.57 (125.18.215.57) 7.855 ms

125.18.215.61 (125.18.215.61) 9.723 ms

125.18.215.57 (125.18.215.57) 7.913 ms

5 116.119.42.61 (116.119.42.61) 42.656 ms

116.119.119.198 (116.119.119.198) 43.526 ms

116.119.42.27 (116.119.42.27) 43.272 ms

6 400940.sgw.equinix.com (27.111.231.33) 44.674 ms 51.554 ms 46.890 ms

7 * * *

8 * * *

9 * * *

10 * * *

11 * * *

12 * * *

13 * * *

14 * * *

15 * * *

16 * * *

17 * * *

18 * *

traceroute to 66.33.22.1 (66.33.22.1), 64 hops max, 40 byte packets

1 192.168.1.1 (192.168.1.1) 3.890 ms 3.724 ms 3.265 ms

2 10.240.9.208 (10.240.9.208) 5.461 ms 5.904 ms 6.579 ms

3 * * *

4 125.18.215.57 (125.18.215.57) 6.264 ms

125.18.215.61 (125.18.215.61) 6.566 ms

125.18.215.57 (125.18.215.57) 10.110 ms

5 116.119.161.51 (116.119.161.51) 57.500 ms

116.119.81.143 (116.119.81.143) 44.676 ms

116.119.161.0 (116.119.161.0) 56.681 ms

6 400940.sgw.equinix.com (27.111.231.33) 42.922 ms 44.104 ms 42.155 ms

7 * * *

8 * * *

9 * * *

10 * * *

11 * * *

12 * * *

13 * * *

14 * * *

15 * * *

16 * * *

17 * * *

18 * * *

19 * * *

20 * * *

21 * * *

22 * * *

23 * * *

24 * * *

25 * * *

26 * * *

27 * * *

28 * * *

29 * * *

30 * * *

31 *

All deployments are facing the same issue

Noted, I have pass this on to our network team. We do have something in flight that should address, we are targeting next week for the fix. The problem is that some ISPs are not keeping the updated peering tables that we have published to get the hops in the right order.

How much customer impact are you facing?

At the moment, this is happening on one of our realtime endpoints (which is not high impact). I'm worried that this might start affecting other high impact API response times. Hope this can be resolved at the earliest.

We're actively working on it, going to subscribe you to notifications as the ticket gets completed.

🛠️ The ticket Global Connectivity Improvement has been marked as todo.

Howdy!
Just heard back from our infra team and we have recently implemented a global backbone. It should improve latency across regions however if there is any application level misconfiguration it won't help with that.

How does your service latency look now?

It isn't resolved. Attaching the http response logs for a specific endpoint. See the upstream response duration ( 1ms). These are from India and should be routed directly to Singapore (where our instances are) but are getting routed through Europe - look at the edge

@noahd Can you please share update on this? This is still not resolved for us.

We had a similar issue recently with networking from Sydney being routed through us-west instead of Singapore.
We have an update in place now which should resolve routing to Singapore.

Are you able to give the services a redeploy and see if its routing as expected?

Not resolved. It is the same. Nothing changed. View the screenshots

Can you provide an update on this

🛠️ The ticket Slow Response Times has been marked as triage.

🛠️ The ticket Slow Web Response Times has been marked as triage.

I'm still awaiting a response/update on this

Hey! Apologies for the lack of update here.

Can you share what that endpoint is doing? Is it keeping connections open?

From what we can see, all other requests to the endpoint (POST, OPTIONS, etc.) is fast and completes within miliseconds.

Only GET /api/realtime is precisely 2min5sec on every request, which indicates it could be something keeping the connection alive and closing it after 2min5sec such as a WebSocket.

✅ The ticket Slow Web Response Times has been marked as completed.

✅ The ticket Global Connectivity Enhancement has been marked as completed.