2 months ago
Can the railway help us identify why we have such high memory usage? We tried to use on local, but no such high memory usage.
Pinned Solution
2 months ago
you can clearly see in the replica view that one replica is way higher than the other (hitting 25gb+) while the second stays lower then the first, i think it's not a global issue, it's one replica leaking memory
since you're on python 3.11, the easiest way to debug which functions eat the most memory is to use tracemalloc, it's built into python so no extra install needed, just add this to your code:
import tracemalloc
tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
print(stat)this will print the top 10 lines in your code consuming the most memory, run it and check your railway logs to see the output
8 Replies
2 months ago
Your MuseAI-Master service accounts for $68.20 in memory and MuseAI-Develop accounts for $21.40 in memory this billing period, making memory 70% of your total usage. Container memory measurement includes all memory allocated by the process, including filesystem caches and buffers, which Linux counts toward the container's cgroup memory - this commonly results in higher reported memory than what you observe locally. You can check the exact memory usage over time in the Metrics tab for each service, and set replica limits on memory to cap what each replica can use.
Status changed to Awaiting User Response Railway • about 2 months ago
Railway
Your MuseAI-Master service accounts for $68.20 in memory and MuseAI-Develop accounts for $21.40 in memory this billing period, making memory 70% of your total usage. Container memory measurement includes all memory allocated by the process, including filesystem caches and buffers, which Linux counts toward the container's cgroup memory - this commonly results in higher reported memory than what you observe locally. You can check the exact memory usage over time in the [Metrics tab](https://docs.railway.com/observability/metrics) for each service, and set [replica limits](https://docs.railway.com/pricing/cost-control) on memory to cap what each replica can use.
2 months ago
How can I debug memory to see more detail of the memory usage?
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 2 months ago
2 months ago
Hello qinbatista
go to your service → metrics tab, you can see your memory usage over time. if you have multiple replicas, switch from "sum" view to "replica" view to see which specific replica is consuming the most memory. the dotted lines on the graph show when each deployment happened so you can see if a specific deploy caused the spike
Hope this help you :)
2 months ago
if you need more help debugin let me know , you can just share a screenshot of your metrics tab so we can see the memory usage over time and spot any spikes?
domehane
if you need more help debugin let me know , you can just share a screenshot of your metrics tab so we can see the memory usage over time and spot any spikes?
2 months ago
Hi Domehane, thank you for your replying, there is a screenshot of the memory usage. I am wondering if I can debug the server to find which functions are using the most memory. It is not common. I tried to deploy on my own Docker on a local machine, but I didn't get that crash memory usage.
Attachments
2 months ago
okay so two things from the screenshot , first you're on "sum" view which combines all replicas together, can you click on "replicas" to see if one specific replica is causing the spikes or if it's all of them equally
second, to debug which functions use the most memory, what language/framework is your app running? the answer depends on your stack (node, python, etc..)
domehane
okay so two things from the screenshot , first you're on "sum" view which combines all replicas together, can you click on "replicas" to see if one specific replica is causing the spikes or if it's all of them equally second, to debug which functions use the most memory, what language/framework is your app running? the answer depends on your stack (node, python, etc..)
2 months ago
we are using python3.11 now, here is screenshot of replicas
Attachments
2 months ago
you can clearly see in the replica view that one replica is way higher than the other (hitting 25gb+) while the second stays lower then the first, i think it's not a global issue, it's one replica leaking memory
since you're on python 3.11, the easiest way to debug which functions eat the most memory is to use tracemalloc, it's built into python so no extra install needed, just add this to your code:
import tracemalloc
tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
print(stat)this will print the top 10 lines in your code consuming the most memory, run it and check your railway logs to see the output
Status changed to Solved chandrika • about 1 month ago