10 days ago
Hello,
Since a few days the memory usage of my backend service has spiked up to a constant 17 to 18GB which seems a lot to me since there's not that much activity on my website.
What could cause such a high memory usage all at once?
Is it necessarily due to an error on my side or could this be normal?
4 Replies
10 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 10 days ago
10 days ago
Check your nestjs code for memory leaks for example global static variables. See if you have --max-old-space-size set anywhere.
Could be a recent update you did to your code
9 days ago
Thanks for the input, I haven't set any node options regarding --max-old-space-size.
Regarding the updates I did to my code, these were minor things with no big changes to the backend so I don't believe it comes from there.
I just compared the CPU and Memory usage over 7 days for the sum and the replicas and as you can see there's some very small difference but what I find troubling is my backend has a volume attached to it so there shouldn't be any difference between the sum and the replicas.
I don't know if this makes sense or if those small variations are totally normal.
I just tried to push a new build without any changes to my code and now the memory usage has just dropped down immediately as you can see, so this makes me point in the direction of an infrastructural issue on Railway side. Don't know if it's a known issue but it would be nice to hear some feedback as this issue resulted in an abnormally high invoice for our app last month.
9 days ago
Looking at the charts, it appears that you have a memory leak in your app such that the memory is not getting released. If you can co-relate to the action performed on the application with the increase in memory.
To debug this you could generate heapdumps and analyse those dumps.
You can search about this on how to generate and inspect, there are a few different ways to do it. If you need specific instructions on how to do this I can post here.
8 days ago
Thank you for the input.
It is indeed a leak, I managed to pin it down to Node's rss memory. It appears the leak comes from Prisma (version 6.19.1), it's completely insane that an ORM has leaks... their communication is also very shady on the subject, seems like they didn't disclose the issue openly...
I found this very interesting post here in case someone encounters the same issue:
https://github.com/prisma/prisma/issues/25371
Apparently the solution would be to switch to Prisma v.7 and upgrade to the new prisma-client provider which uses the new Rust-free client (the Rust client being the one that leaks).
I'll post the outcome here if I manage to mitigate the leaks with the upgrade and for the time being I'll simply redeploy my service every now and then to keep the memory usage low.