8bccf693-4059-4ef3-9dd0-55493979fdb7

I should note that it does get stuck there and will not go down again

are you loading the file into memory?

and, reading a 30gb file from the disk to where?

my rust backend need to read all 180k file I have the attached volume with the service. They're about 170kb each

locally it goes fine and doesn't fill up my ram

right but you are reading these files off of disk? where are they going beside into ram?

I'm not sure to understand the question

I'm reading like literally at /cache/something/somefile.mspack and deserializing it, doing compute and drop

/cache is where my volume is mounted at

so these files are loaded into memory then, I'm not sure why you are surprised that memory has increased?

once I have read and made use of the information how can I discard and let the unused ram go back ?

locally it is discarded right after each read, so my ram never go up. I can run this program with 1gb or less ram

that wouldn't be a platform specific question

Well if that doesn't sound like a bug/misuse to you, then I got my answer I guess 😄

yes unfortunately this would be a code issue

Yep, you will need to deallocate that in Rust

sounds like you aren't resolving your lifetimes afaik

I tried all I could think of really 😕 even with an explicit drop(variable_with_deserialized_data) at the end of each loop. Even after my job is done and clean it doesn't drop. I still want to precise that locally the same job never goes above a few mb of ram, on the same dataset.

nixpacks or dockerfile?

legacy or v2 runtime?

[build]
builder = "NIXPACKS"

[deploy]
startCommand = "./api"
healthcheckPath = "/health"
healthcheckTimeout = 100

[phases.setup]
nixpkgsArchive = 'a459b363de387c078080b719b30c54d8a79b4a3e'
nixPkgs = ["...", "ffmpeg"]
nixLibs = ["...", "dav1d"]

not sure that matter tho, also I do my build on github action because they're not trivial and I do a final railway up -e {env} -s {service_id} to upload

thats going to cause railway to run another build

are you sure you're did not mix 2 problems 😄 I was on something unrelated to deployment, I don't have deployment problem

i know, im getting side tracked, just wanted to point it out

but when in doubt write a dockerfile that uses alpine instead of nixpacks

have seen alpine based dockerfiles help with strange memory issues before plenty of times

I could give a try

cant hurt

unfortunately did almost everything but for alpine I need some special target compilation and I can't get it to work easily with my rust code. got too much deps

I could try a docker image on ubuntu or something but at this point we're back to what nixpack does

ah yes the joys of rust, compiling

so I did a small experimentation, last peak is me running that heavy task (limited to 20k item to not go to 30gb again). I have changed the start command to while true; do free -h; sleep 5; done & ./api and here are the result. We can clearly see that the used column stay to 251gb, before and after the tasks (why does it show me the host machine spec's instead of my docker, no clue). But the column buff/cache is billed on me and indeed grew by a few gb.

buffer / cache is indeed included in the results of docker stats

I had some hope for a moment with O_DIRECT, it works locally, doesn't bump the buff/cache but on railway I get an error, I guess the filesystem doesn't accept this custom flag <:sossa:756985575243776191>

this is pain

I wonder if the v2 runtime would also gather metrics that include the buffer / cache, but you would need to find a way to run your tests without a volume since any service with a volume defaults back to the legacy runtime despite the selector saying v2

Have you tried copying the volume’s contents into the container’s disk on startup?

In that case you may be able to do local file system file streaming in rust and it shouldn’t increase the memory

Local file system is a little different than docker mounted volumes i think

volumes are ext4 on legacy, and zfs on v2

I forgot railway doesn’t actually use docker

legacy runtime does

Man I don’t even use railway

🗣️

v2 uses podman

Man portable air defense?

Oh wait that’s manpad

Since I could not find a solution with railway for this particular thing, I bought a server somewhere else, although the disk io isn't as good as railway and this add me some complexity for deployment and monitoring :( But yeah I can't afford adding 200$+ to my monthly billing just to read a few files sometimes. I still use railway for a lot of other stuff and I'm happy with those. I just figured out that I should railway where it is helping me instead of trying to fight it. No hate, I can understand why buff/cached is counted. Just wanted to give an update for future people searching this thread.

i wrote a benchmark test (in Go) to write 30000 1MiB files (for a total of 30GiB) to disk at 250 files concurrently.

running this locally of course there was no wild increase in memory.

running this program on railway with the legacy runtime had my memory reach ~23GB.

i then switched to the v2 runtime and ran the program again and the memory never increased above ~45MB, it also wrote the data a bit faster.

i then re-ran the test on the v2 runtime but this time configured it to write 50GIB worth of files and still saw no wild increase in memory.

and just to be sure one last time, i switched back to the legacy runtime and ran the test to write the 50GIB worth of files (same concurrency), and the memory this time peaked at 32GB.

tl;dr this issue is fixed in the v2 runtime but only the legacy runtime has support for volumes (even if you select the v2 runtime and you have a volume it will run with the legacy runtime)

heres a memory graph that backs up these statements

yes I know I'm writing files instead of reading them, but the same issue is being surfaced

interesting

I need a sticker "V2 fixes it"

v2 builder / runtime / proxy has just simply fixed a lot of issue thus far

when is v2 going ga ?

asap once the bugs are fixed

nice

from the sounds of it the legacy runtime will not be running on bare metal

it sounds like bare metal will be v2 runtime exclusive

oh nice

We are taking our time btw

We aren’t going to V2 cutover until everything is polished

we aren't going to remove the legacy option*

Ofc we are moving fast- but migrating running workloads over will take time

Its likely that we have a one way migration and will remove the option, like we did for BuildPacks to NixPacks

thankfully I wasn't around for that 😆

It was very painful

But we did it

And we’ll accomplish this as well

I just wanna know if we will see V2 runtime support volumes before bare metal, it's whats stopping alaanor from running this project on railway

Ofc, this will be added before metal

Completely stateless workloads aren’t very useful

char said otherwise, unless I was misunderstanding him

Metal workloads will mostly be Trial and experimental workloads until we flight Railway Metal that requires volunes

If the servers are plugged in, why not serve from then

But we won’t stop the metal rollout until we have vol. support

right but people could benefit from volume support on the v2 runtime with the current gcp hosts, like OP, or everyone running uptime kuma that are getting EHOSTUNREACH

Yea, heard, it’s a, we are speedrunning all short coming fixes when we can

Never enough hands on boards

But its not a OR its a AND

Metal is happening on a different timeframe than V2 cutover

It just so happens that Metal will be V2 only, no sense in extending the lifetime of Legacy

ay at least i was right in that regard

either way, would you say it would be safe to mark this thread as solved, and would i be correct in saying this wont be getting fixed on the legacy runtime?

Yep

This is really cool, appreciate the finding a lot. Thanks 👍 I'll be checking the railway changelog for v2 with volume frequently and hopefully one day I can be fully back on railway :)

hopefully!

@Brody @angelo I saw v2 volume are now a thing and so I spent the day to setup the stuff and try on railway again but unfortunately it's still stuck this way. Not to complain or anything, just wanted to share that it did not magically fix that, as we though it perhaps would.

you might not be on the v2 runtime

at least the UI said I was on it, I remember you said that it might be a lie from the UI because it would not works with volume, but now we got volume on v2 so I though I could maybe trust that UI

not all of railway's hosts support volumes on the v2 runtime, a surefire way to be sure you are on the v2 runtime would be to check for container event logs like "starting container"

Just wanted to tell that I have finally moved back this particular service to railway and it's working great now 👍 thanks again for all

that's awesome!