I can grow this to 10gb for you when I'm back at the computer.

to download files you would need a service that doesn't write data to the volume on startup, filebrowser writes it disk based database to disk along with metadata

Alright, that would be awesome, let me know when you are able to do it!

Is there any existing template for that? Otherwise I will write something myself in case of emergency in the future like that

nothing that I'm aware of to simply dump your volume in one go like filebrowser could

done

thank you, this is very much appreciated!

And I'm writing a stupid simple template to do a dump of a volume

like just dump a zip?

it sure is a good thing we have volume alerting on the pro plan 😉

yep, a zip

Sure is, but tbh, an automatic scaling/resizing would be even better. Alerts can still be missed

thats why you can set alerts for different thresholds 🙂

I also need to look better into the sqld configuration, I might be able to reduce the frequency of WAL so it doesn't grow so fast

i mean, Pro allows you to grow your own volume to 250GB so take that how you want haha

Ahah yeah, I wouldn't have to worry that much. I'm still following closely the changes on the Pro plan, with maybe included usage in the future.

For the moment, for my side-project the hobby plan is enough

sounds good to me, just a fair warning, your next volume increase will need to be done by upgrading to pro.

Alright, that should do the job https://railway.app/template/EBwdAh

are you down for some template feedback?

Yeah sure, it’s the first one I wrote

your code expects user to mount the volume to the correct location.

if you give the user room to do something wrong, it's guaranteed that they will do something wrong.

instead have the code use that railway volume mount path variable so that there is no opportunity for the user to do something wrong.

another question, how fast can node even zip for example a 50gb directory?

I actually started using a variable for the volume path but when mounting the volume to the service, you have to define the mounting path, so it’s still error prone. Unless you can define the volume path when mounting it to your service?

Good question, it might not be the best. Do you have a way for me to try with some dummy data through Railway. I could write the zipping part in whatever is fastest

if you use the variable, it doesn't matter where the user mounts the volume

as for benchmarking node zipping a directory, it doesn't need to be railway specific, you can run the test with a 50gb directory on your own computer as long as you have an nvme drive

Alright cool, I'll improve that, thanks 👍

alright, updated the template, rewrote in GO, perfs are quite good locally, ~1:30min for 40GB. Through Railway I only have 400MB of data to play with for testing, and that took 20sec.

If you have any dummy big data to try it out, could be interesting. I have one issue to fix, where the memory stays very high after streaming the ZIP, will look at it later

haha if i knew you where gonna rewrite it in go i would have given you some tips

i know go's gzip package is single threaded, so im going to assume so is it's zip package, you could swap to using

I am using this package https://pkg.go.dev/github.com/klauspost/compress/zip from the same user, can give a try to the one you mentionned

looks way faster with pgzip

how much faster are we talking?