6 months ago
We've been working on migrating one of our projects from Heroku to Railway. For preliminary tests we've been using an internal variant of the product for light use. The costs were greatly reduced, and the internal performance appears to have slightly improved.
However, I've been experiencing some pretty bad network latency. Sentry tells me that a specific endpoint takes an average of 90ms to process. However client-server-client request time for the same endpoint takes anywhere between 300ms and 500ms from Montreal to US West Metal. Request times were similar from Montreal to US East Legacy. They are also similar between the railway-provided domain and our custom domain. And finally with or without Metal Edge. On Heroku, these same requests would take nearly the same time as the internal processing time, which is anywhere between 100ms and 150ms for the US Virginia instance.
Pings from my CLI gives the following results. I'm no expert in networking but the results aren't particularly terrible to me.
PING XXXXXXXXX: 56 data bytes
64 bytes from XXXXXXXXX: icmpseq=0 ttl=106 time=33.650 ms 64 bytes from XXXXXXXXX: icmpseq=1 ttl=106 time=32.016 ms
64 bytes from XXXXXXXXX: icmpseq=2 ttl=106 time=40.977 ms 64 bytes from XXXXXXXXX: icmpseq=3 ttl=106 time=25.606 ms
64 bytes from XXXXXXXXX: icmpseq=4 ttl=106 time=34.603 ms 64 bytes from XXXXXXXXX: icmpseq=5 ttl=106 time=29.346 ms
Am I missing something?
0 Replies
If a staff member is looking at this, the specific endpoint being tested is /api/emailinstance
Please let me know if there's any additional information I can add to help my case.
My requests from Montreal to US West Metal are getting routed through Amsterdam this morning.
Also, when I lookup the location of the (non-static) IP address associated to my server, it shows the server is located in US Virginia. It's configured to be hosted in US West.
IP Details For: XXXXXXX
Decimal:601120354
Hostname:XXXXXXXXXXXX.bc.googleusercontent.com
ASN:15169
ISP:Google LLC
Services:Datacenter
Country:United States
State/Region:Virginia
City:Ashburn
Latitude:39.0395 (39° 2′ 22.11″ N)
Longitude:-77.4918 (77° 29′ 30.50″ W)> traceroute XXXXXXXXX
traceroute to XXXXXXXXX (XXXXXXXXX), 64 hops max, 40 byte packets
1 rt-ax58u-71a0 (192.168.50.1) 1.872 ms 1.456 ms 1.289 ms
2 192.168.0.1 (192.168.0.1) 2.604 ms 2.418 ms 3.717 ms
3 modemcableXXXXXXXXX (XXXXXXXXX) 17.087 ms 16.971 ms 19.234 ms
4 XXXXXXXXX (XXXXXXXXX) 18.594 ms 26.007 ms 18.487 ms
5 motl-b2-link.ip.twelve99.net (213.248.86.238) 15.934 ms 15.877 ms 15.092 ms
6 * nyk-bb5-link.ip.twelve99.net (62.115.137.142) 25.988 ms 23.839 ms
7 ldn-bb1-link.ip.twelve99.net (62.115.139.245) 98.385 ms 94.237 ms *
8 * adm-bb1-link.ip.twelve99.net (62.115.139.144) 108.336 ms 101.517 ms
9 adm-b4-link.ip.twelve99.net (62.115.136.35) 104.051 ms
adm-b10-link.ip.twelve99.net (62.115.120.227) 97.446 ms
adm-b4-link.ip.twelve99.net (62.115.136.35) 104.536 ms
10 4.184.214.35.bc.googleusercontent.com (35.214.184.4) 133.684 ms
google-ic-339903.ip.twelve99-cust.net (62.115.167.225) 142.598 ms 132.437 ms6 months ago
hello, we have made some changes to our anycast routing, can you please check if you are now being routed to our us-east region? -
6 months ago
I'm being routed to us west…
6 months ago
I'm Ontario
6 months ago
Not sure if relevant, just chiming in in case it is
6 months ago
we live like an hour away from each other, so thats not ideal
6 months ago
for reference, i am being routed to us-east
6 months ago
Hm very weird
6 months ago
gonna ask you for some testing in #🦸｜conductor-chat
Yeah the changes to routing helped.
However the f***up was on my part. I didn't configure gunicorn to run with a defined amount of workers and I'm pretty sure it defaulted to just 1. Simultaneous requests were getting queued one after the other. So even if a request took 100ms to complete, it had to wait for its turn before being processed by the server so in reality it would sometimes take thousands of millisecs.
6 months ago
iirc gunicorn defaults to 3 sync workers, absolutely horrible defaults for anything with anywhere near modest or concurrent traffic