a year ago
Hi,
I'm trying to understand what factors are at play which might explain the disappointing performance I'm seeing with my django app. I'm testing with an endpoint which does 5 basic SQL queries and returns less than 1kb of data.
I am doing a simple load test, calling my endpoint 20 times with one user. On railway, my average response time is 800ms, which is pretty slow, and I figure maybe this is due to network hops (my database is in AWS us-east-1, it looks like railway is in GCP us-west?). The same endpoint run locally against the same DB takes ~200ms.
With 2 concurrent users, the average response time jumps to 1500ms, and at 4 users, it jumps to 2800ms, while locally this is 230ms. During these tests, I've never been able to make the CPU usage of my app move beyond 0.1 vCPUs, so there's clearly a bottleneck somewhere. What other possible issues should I look into here?
5 Replies
a year ago
first, I would run the database on Railway instead and connect to it via the private network, or at the absolute minimum, move the aws database to us-west too.
second, this increase in latency with added concurrent requests is likely due to gunicorn's default sync workers, I would look into adding more workers and threads, this is a great resource https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7
Status changed to Solved railway[bot] • about 1 year ago
a year ago
Ahh, the gunicorn setting was exactly what I was missing, thank you! I updated to 10 workers and am seeing much better utilization. Is there a reason we don't set workers=2*vCPU or similar by default?
My database is shared by some other projects, so I think I will just eat the latency for the extra networking for now. If this project becomes more popular I will move it to us-west (or move this app to east)
a year ago
whatever gunicorn's defaults are, are going to the the defaults when running gunicorn on railway, and for some reason gunicorn runs I think two sync workers? up to the user to configure gunicorn for their specific work loads.
also, please keep in mind that by running a database outside of railway's network (and subsequently not connecting to it via the private network) you not only incur more latency, but you will incur outbound network fees from your railway service communicating with a database over the public network, just something to keep in mind.
a year ago
Thanks for the heads up about the db. Really appreciate your fast responses to all my questions here.
And I understand the decision to use gunicorn's defaults for new apps. I might suggest making this more obvious in the docs though, since it seems necessary to add more workers in order to make use of all the hardware a user is paying for. Maybe this is a simple thing that most python devs know to do, but as someone unfamiliar with django hosting I wasted a lot of time profiling my app trying to understand what was wrong.