2 months ago
When traffic is high RAM consumption is 5GB which is not freed up when traffic is low. I've tested it, it's not application level issue but platform issue when OS doesnot free up unused memory.
So todo this i usually have to restart it again, so it consumes less space then.
Anyone have the solution to this?
Pinned Solution
2 months ago
MinIO uses GO language. Go runtime holds allocated memory to avoid the overhead of frequent re-allocation, combined with MinIO's aggressive caching. So you can try below 2 options.
- Add below in environment variables
GOMEMLIMIT=2048MiB # Adjust based on your plan or memory limit GOGC=50 # The default is 100. Lowering it to 50 makes GC run twice GODEBUG=madvdontneed=1 # This tells the runtime to notify the OS that it can take back the memory pages immediately after they are cleared - Use https://railway.com/deploy/1FHSG9 template to restart when traffic is low. This can be helpful if your traffic varies on daily basis.
8 Replies
2 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open brody • 2 months ago
ian_rosenfeldt
It's MINIO storage.
2 months ago
Hello,
i think minio keeps memory buffers after traffic drops and reuses them for future operations, so ram doesn't go back down automatically
the official fix from the minio team is to set this env variable to make the go garbage collector more aggressive:
GOGC=10
add it to your railway service environment variables and redeploy, no restart loop needed after that
2 months ago
GOGC=10 is a go (since minio is written in go) environment variable that makes the garbage collector more aggressive
2 months ago
MinIO uses GO language. Go runtime holds allocated memory to avoid the overhead of frequent re-allocation, combined with MinIO's aggressive caching. So you can try below 2 options.
- Add below in environment variables
GOMEMLIMIT=2048MiB # Adjust based on your plan or memory limit GOGC=50 # The default is 100. Lowering it to 50 makes GC run twice GODEBUG=madvdontneed=1 # This tells the runtime to notify the OS that it can take back the memory pages immediately after they are cleared - Use https://railway.com/deploy/1FHSG9 template to restart when traffic is low. This can be helpful if your traffic varies on daily basis.
bilalnawaz072
MinIO uses GO language. Go runtime holds allocated memory to avoid the overhead of frequent re-allocation, combined with MinIO's aggressive caching. So you can try below 2 options. 1. Add below in environment variables `GOMEMLIMIT=2048MiB # Adjust based on your plan or memory limit GOGC=50 # The default is 100. Lowering it to 50 makes GC run twice GODEBUG=madvdontneed=1 # This tells the runtime to notify the OS that it can take back the memory pages immediately after they are cleared` 2. Use <https://railway.com/deploy/1FHSG9> template to restart when traffic is low. This can be helpful if your traffic varies on daily basis.
2 months ago
for GOGC=50 i think its too conservative, so the default is already 100, dropping to 50 only makes gc run twice as often. for a case where ram is sitting at 5gb idle, GOGC=10 is more appropriate and is actually what minio team members have suggested specifically for this type of issue , also for GOMEMLIMIT=2048MiBmuhammad_iqbal's service already peaks at 5gb under high traffic, so capping at 2gb could cause performance issues or even oom kills during peak load
bilalnawaz072
MinIO uses GO language. Go runtime holds allocated memory to avoid the overhead of frequent re-allocation, combined with MinIO's aggressive caching. So you can try below 2 options. 1. Add below in environment variables `GOMEMLIMIT=2048MiB # Adjust based on your plan or memory limit GOGC=50 # The default is 100. Lowering it to 50 makes GC run twice GODEBUG=madvdontneed=1 # This tells the runtime to notify the OS that it can take back the memory pages immediately after they are cleared` 2. Use <https://railway.com/deploy/1FHSG9> template to restart when traffic is low. This can be helpful if your traffic varies on daily basis.
2 months ago
Can you set a preferred bounty workspace so we can award your bounty?
Status changed to Awaiting User Response Railway • 2 months ago
sam-a
Can you set a preferred bounty workspace so we can award your bounty?
2 months ago
Done!
Status changed to Awaiting Railway Response Railway • 2 months ago
Status changed to Solved chandrika • 2 months ago