Hey there! We've found the following might help you get unblocked faster:

• 🧵 After Celery deployment, memory usage and network egress has gone up.

• 🧵 Simple HTML high memory usage

• 🧵 Database Memory Usage Unusually High

• 🧵 Firecrawl template high memory usage

If you find the answer from one of these, please let us know by solving the thread!

Hello,

The majority of costs are coming from memory, which is used by your Celery worker.

If you would like to reduce costs, you may want to look into a more memory-efficient solution to handle what you are using Celery for.

Best,
Brody

Have you got any refernce for a more memory efficient solution?

I don't, but I'm sure the community can give you some advise, would you like me to open this up to the community?

sure, that's fine

Have you viewed this?

https://docs.celeryq.dev/en/latest/userguide/optimizing.html

They seem to have some tips on optimization. Unfortunately, if that doesn't work, that's likely just the nature of Celery. I wouldn't expect the jump from 10-100 users to be as large as 0-10.

I can't help on the memory optimization, but can you trying limiting the memory and see if your app still work with less memory? is not a real solution but it might give you more time to find the real solution....

Current costs arent really the problem because we're still on dev stage, Im afraid of the costs of scaling our product when we finally deliver to our client that has ~400 interested users.

I've checked this out and came to some conclusions, I need to change worker prefetch multiplier to 1 to avoid task hoarding. But the most important that I dont know if that's even possible to do on railway is check if it supports autoscaling based on Queue depth in Redis, CPU/memory metrics. Is that possible? Or if it doesn't support the "scale to zero", use a cron job to check queue depth and start/stop the worker service or move to a platform that supports serverless workers (AWS Lambda, Google Cloud Run).

Railway supports serverless, there is a toggle at the bottom of the service config. Other than that, auto scaling is pretty limited. You can create replicas, but I'm pretty sure they all run.

I've checked serverless docs, and as the connection between celery and database is always on, It wont work for what I need, and it will be always in idle

how many workers do you have? I'm not expert on celery, I did a quick research (I'm not gonna lie, all with ai), the suggestions they gave is to have less workers and try with --pool=threads ...if you are using prefork you will use a lot of memory...but it really depends on the kind of job you need....

you can ignore this if it doesn't make sense, I'm just throwing more ideas in the bucket

Hmm, Im already running 2 workers (down from 8), and I'm thinking about switching to cron mode (5 min schedule) with webhook triggers for on-demand execution, so threads would only add very low extra savings but I'll consider it if I need more optimization later. The downside of the webhook+cron approach is that this might add 30s delay to my calls to worker at least.