5d77fb37-1fba-4a55-bbd3-e0fab8243723

Network latency? Quick responses are loading slowly

this is for the threaded service in the prod env?

Yes

@Brody sorry for being slow. It's good to see you again.

Here's some HTTP logs from today. I couldn't find them in the regular log search (is there a way to see them there?).

The response times are showing up as really slow.

first thing im noticing is that you are running gunicorn with its default sync workers, the sync workers really arent suitable for production use, can you try switching over to gunicorn with uvicorn event workers, and start with 4 workers

I released an update to use the uvicorn workers. They seem to be working fine, but we're getting an intermittent error. I'm wondering if it could be proxy related?

OperationalError
connection failed: c120:567d::4d:c3b5:bacd", port 5432 failed: FATAL:  sorry, too many clients already
connection to server at "fd12:c120:567d::4d:c3b5:bacd", port 5432 failed: FATAL:  sorry, too many clients already

Things are quiet on the database side.

how many workers are you running now

4

The release is clear in the memory chart

Here's a closer view

looks like you are connecting to the database for every thread that is spawned then, you would have to rethink your database client a bit then

I'm using the default django one. It sounds like connection pooling is a natural next step.

yeah i think some re-architecturing is going to be required so that you can move away from the slow sync workers

Thanks. I'm new-ish to this type of change. Any pointers are apppreciated.

Find a way to open a single pool of databases for the global process, right now you are opening a pool for every thread.

I found a pool option for postgres in Django that seems worth a try.

    "OPTIONS": {
            "pool": True,
        },

https://docs.djangoproject.com/en/5.1/ref/databases/

Can't hurt to try