6 months ago
I'm deployed on railway metal (Amsterdam). Backend is on api.zennoia.com, and proxied behind Cloudflare (Free Plan) with SSL/TLS set to Full (Strict) if that's relevant. Cloudflare is also used for DNS.
Mumbai<>Amsterdam could be 180ms (theoretical best as per https://wondernetwork.com/pings/Mumbai), and I was investigating my API's response times to get as close as possible to that figure.
Have run some basic testing with and without cloudflare proxy with a public unauthenticated endpoint https://zennoia.com/startups/zennoia (smash refresh button in Chrome incognito with cache disabled), and I'm trying to investigate why either of them show api.zennoia.com server response times in the network tab anywhere in the range of 300ms to 600ms on an average, with >1s round trip times seen quite often too.
With Cloudflare proxy enabled, I see the round trip times swing a LOT, often in the range of 300-800ms, and very often it would swing as high as 1s, outliers even higher. On a good day 200ms feels spectacular, but I don't get to see that often. Usually it's averaging around 600ms for cold starts.
With Cloudflare proxy disabled (Cloudflare for DNS only), I see far higher average response times in the range of 400-800ms average, with the outliers going upto 1s but not higher. Slightly more consistent than with CF proxy, but takes awfully longer.
All of the above testing is as mentioned, in an incognito tab where it's creating a fresh connection. But in the app (investors.zennoia.com) where the connection to the API is warmed up, with the proxy it generally takes 200-300ms, and without the proxy 200ms-400ms.
I'm not looking to explore caching mechanisms or band-aids for this problem, but simply attempting to understand if this is a consequence of Railways network path, Cloudflare's network path, or has something to do with my network provider (Reliance Jio), but I've asked a friend to run the same tests from Canada and they were seeing ~600ms numbers with CF proxy disabled.
I've added a logger to my backend and requests are being processed in the range of 0ms (no database calls) to 6ms (with database calls), so the problem is narrowed down to the network layer.
Please find a traceroute.txt file attached which has the outputs of "traceroute api.zennoia.com" command run with and without cloudflare proxy. Please let me know if there's any more information required for the same.
Thanks,
Aayush Shah
aayush@zennoia.com
Attachments
8 Replies
6 months ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
6 months ago
This seems to fall within the expected range per the page you provided (280ms average). What's your expectation for the latency? Is it the spikiness that's the issue, or are you looking to achieve the fastest theoretical speeds possible?
Status changed to Awaiting User Response Railway • 6 months ago
6 months ago
Hi Ray, I see that on https://wondernetwork.com/pings/Mumbai/Amsterdam at the time I was running my tests it was around 180ms maximum and 150ms average, and at the time of your checking it had deteriorated to 280ms average. For my considerations I was using 180ms max figure as a reference while running the tests.
I want to bring it as close to the average, and reduce the very high variance seen right now.
The numbers I was seeing for Mumbai (me) <>Amsterdam (railway deployment) at the time of filing the ticket:
Mumbai-Amsterdam: 150ms average, 180ms max ping times
First connection/query
With Cloudflare proxy average: >1s for 1 round trip, with huge, frequent outliers averaging as high as 4-5s
Without Cloudflare proxy average: ~800ms for 1 round trip, occasional outliers averaging as high as 1s
Subsequent queries on a warmed up URL
With Cloudflare proxy average: ~250ms for 1 round trip, occasional outliers averaging as high as 600ms
Without Cloudflare proxy average: ~350ms for 1 round trip, occasional outliers averaging as high as 800ms
I recently changed the SSL/TLS to "Full" as recommended on https://docs.railway.com/guides/public-networking#provider-specific-instructions but still don't see any improvements.
And at the time of writing this comment, Mumbai<>Amsterdam average is 127ms, and with Cloudflare Proxy disabled and SSL/TLS set to "Full", I still continue to see the ~350ms with the occasional ~800ms outliers on subsequent queries on a warmed URL.
Preferred outcome of this ticket: Get reliable/consistent round trip times to api.zennoia.com behind a cloudflare proxy as close to the average seen between Mumbai-Amsterdam
Status changed to Awaiting Railway Response Railway • 6 months ago
Status changed to Awaiting User Response Railway • 6 months ago
6 months ago
Without Cloudflare Proxy:
aayush@Aayushs-MacBook-Pro zennoia-monorepo % traceroute api.zennoia.com
traceroute to lywyrsft.up.railway.app (66.33.22.142), 64 hops max, 40 byte packets
1 reliance.reliance (192.168.29.1) 3.900 ms 4.963 ms 3.640 ms
2 10.37.48.1 (10.37.48.1) 5.264 ms 5.041 ms 6.601 ms
3 172.31.0.254 (172.31.0.254) 8.361 ms
172.31.0.236 (172.31.0.236) 12.613 ms
172.31.0.254 (172.31.0.254) 14.163 ms
4 192.168.70.14 (192.168.70.14) 13.747 ms
192.168.70.16 (192.168.70.16) 12.316 ms
192.168.70.14 (192.168.70.14) 11.384 ms
5 172.26.76.167 (172.26.76.167) 6.776 ms 5.942 ms 3.593 ms
6 172.26.76.130 (172.26.76.130) 7.245 ms 6.443 ms 6.588 ms
7 192.168.7.250 (192.168.7.250) 6.603 ms
192.168.7.244 (192.168.7.244) 7.514 ms
192.168.7.250 (192.168.7.250) 9.202 ms
8 * * *
9 * * *
10 103.198.140.64 (103.198.140.64) 31.443 ms
103.198.140.170 (103.198.140.170) 28.311 ms
103.198.140.64 (103.198.140.64) 27.428 ms
11 103.198.140.89 (103.198.140.89) 60.167 ms
103.198.140.209 (103.198.140.209) 66.848 ms
103.198.140.247 (103.198.140.247) 60.955 ms
12 103.198.140.207 (103.198.140.207) 110.112 ms
400940.sgw.equinix.com (27.111.231.33) 59.941 ms 60.322 ms
13 400940.sgw.equinix.com (27.111.231.33) 64.426 ms 59.542 ms *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
51 * * *
52 * * *
53 * * *
54 * * *
55 * * *
56 * * *
57 * * *
58 * * *
59 * * *
60 * * *
61 * * *
62 * * *
63 * * *
64 * * *
Status changed to Awaiting Railway Response Railway • 6 months ago
6 months ago
With cloudflare proxy:
aayush@Aayushs-MacBook-Pro zennoia-monorepo % traceroute api.zennoia.com
traceroute: Warning: api.zennoia.com has multiple addresses; using 104.21.88.157
traceroute to api.zennoia.com (104.21.88.157), 64 hops max, 40 byte packets
1 reliance.reliance (192.168.29.1) 8.666 ms 3.622 ms 3.245 ms
2 10.37.48.1 (10.37.48.1) 5.147 ms 5.512 ms 5.111 ms
3 172.31.0.236 (172.31.0.236) 9.074 ms
172.31.0.254 (172.31.0.254) 7.669 ms
172.31.0.236 (172.31.0.236) 6.024 ms
4 192.168.70.16 (192.168.70.16) 5.884 ms
192.168.70.14 (192.168.70.14) 5.531 ms
192.168.70.16 (192.168.70.16) 6.415 ms
5 172.26.76.167 (172.26.76.167) 8.110 ms 4.760 ms 5.259 ms
6 172.26.76.131 (172.26.76.131) 5.007 ms
172.26.76.130 (172.26.76.130) 7.335 ms
172.26.76.131 (172.26.76.131) 6.313 ms
7 192.168.7.248 (192.168.7.248) 9.519 ms 5.687 ms
192.168.7.250 (192.168.7.250) 5.993 ms
8 * * *
9 * * *
10 * 49.44.18.38 (49.44.18.38) 13.234 ms 7.873 ms
11 * 49.44.18.38 (49.44.18.38) 14.796 ms 8.508 ms
12 * * *
13 103.198.140.164 (103.198.140.164) 11.884 ms 6.671 ms 7.606 ms
14 103.198.140.98 (103.198.140.98) 126.989 ms 129.198 ms 129.999 ms
15 de-cix-frankfurt.as13335.net (80.81.194.180) 132.190 ms * 143.942 ms
16 162.158.84.183 (162.158.84.183) 150.557 ms
162.158.84.149 (162.158.84.149) 129.981 ms *
17 104.21.88.157 (104.21.88.157) 134.471 ms 140.798 ms 144.184 ms
6 months ago
Hello!
We're acknowledging your issue and attaching a ticket to this thread.
We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.
Please reply to this thread if you have any questions!
6 months ago
I added a ticket here. We will look at it when we can and close it out when possible. Cheers
Status changed to Awaiting User Response Railway • 6 months ago
5 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 5 months ago