7 months ago
I’m running into an issue with my app on Railway where memory usage keeps going up and never drops. The app downloads files and writes them to /data, which is a mounted volume. When I check the Railway charts, I can see the volume size growing during downloads — which makes sense — but memory usage increases at the same time and just stays high.
I’ve tested this on a few other setups (Raspberry Pi, Intel mini PC, and a Windows machine), and everywhere else memory usage goes back down after the download finishes. The app itself doesn’t seem to be leaking memory — I’ve added logs to report memory usage, and those numbers stay stable even as the Railway memory chart keeps going up.
So I’m wondering if this might be related to how Docker or volumes work on Railway. Maybe something about how IO is cached or how memory is measured? Could it be that Linux isn’t reclaiming cached memory because there’s no pressure?
Have you seen something like this before or can you tell me what Railway includes in its memory charts? So that I can try to debug each metric included in the memory chart. Thanks a lot.
7 Replies
7 months ago
First things first, I would like to know what you're using to download the files. The one thing I can suspect that's going on is you're trying to download a big file and you're not optimizing your download logic such as employing chunking, which in this case, you are hitting an OOM due to the file being too big to reside on memory.
7 months ago
Thanks for your reply. I am using ytdlp to download files. The files can range from megabytes to gigabytes but i dont think it matters because ytdlp does download in chunks and the app has 8GB of memory. I call the binary of ytdlp with a .net process and tell it to safe the file to a temp file. The temp file is written to the volume and converted to a new file on the volume. The temp file is deleted afterwards. Locally i tracked modified files in the docker container and there where no lingering files except in the mounted volume. Temporarely the memory can and should increase because i load a lot of data in memory but it should decrease again as it does anywhere else except on railway. https://github.com/byCrookie/discord-music/blob/main/src%2FDiscordMusic.Core%2FYouTube%2FYouTubeDownload.cs
7 months ago
Thanks for your reply, now let's try profiling your application and see where things are causing issues. Remember, you can SSH into your Railway app, so, if possible, run some profiling there, and see if you can find anything that causes issues.
bycrookie
Thanks for your reply. I am using ytdlp to download files. The files can range from megabytes to gigabytes but i dont think it matters because ytdlp does download in chunks and the app has 8GB of memory. I call the binary of ytdlp with a .net process and tell it to safe the file to a temp file. The temp file is written to the volume and converted to a new file on the volume. The temp file is deleted afterwards. Locally i tracked modified files in the docker container and there where no lingering files except in the mounted volume. Temporarely the memory can and should increase because i load a lot of data in memory but it should decrease again as it does anywhere else except on railway. https://github.com/byCrookie/discord-music/blob/main/src%2FDiscordMusic.Core%2FYouTube%2FYouTubeDownload.cs
7 months ago
Just wanted to pop in here to let you know that ytdl breaks our fair use agreement, please find somewhere else to run ytdl otherwise you are putting yourself at risk of an immediate ban.
Status changed to Solved brody • 6 months ago