Unable to reclaim the stale blocks in a volume

a month ago

I am facing similar issue as of: https://station.railway.com/questions/disk-usage-redis-volume-continuously-inc-510f373f

I tried running fstrim /data but the operation was rejected with an error:

fstrim: /data: FITRIM ioctl failed: Operation not permitted

Since my Redis volume is growing steadily from ~260 MB to ~380 MB over the last month.

Solved

7 Replies

Railway
BOT

a month ago

The fstrim failure ("Operation not permitted") is expected because containers do not have the privilege required for that operation. Volumes are mounted with continuous discard, so freed blocks are returned to the underlying storage layer over time automatically, but the reported usage settles above what df/du show inside the container due to a 16 KB block allocation floor (the storage allocates in 16 KB blocks while ext4 frees in 4 KB blocks, so partially-freed blocks cannot be reclaimed). The only way to fully reset a volume's footprint is to migrate the data to a fresh volume and delete the old one.


Status changed to Awaiting User Response Railway • 28 days ago


Railway

The `fstrim` failure ("Operation not permitted") is expected because containers do not have the privilege required for that operation. Volumes are mounted with continuous discard, so freed blocks are returned to the underlying storage layer over time automatically, but the reported usage settles above what `df`/`du` show inside the container due to a 16 KB block allocation floor (the storage allocates in 16 KB blocks while ext4 frees in 4 KB blocks, so partially-freed blocks cannot be reclaimed). The only way to fully reset a volume's footprint is to migrate the data to a fresh volume and delete the old one.

a month ago

Migrating your data to a fresh volume is the only way to reset the footprint.

Doing this continuously is definitely not a feasible option for the long term.

Volumes are mounted with continuous discard, so freed blocks are returned to the underlying storage layer over time automatically, but the reported usage settles above what df/du show inside the container due to a 16 KB block allocation floor (the storage allocates in 16 KB blocks while ext4 frees in 4 KB blocks, so partially-freed blocks cannot be reclaimed

Is there any particular threshold limit after which it automatically tried to free up the blocked space?


Status changed to Awaiting Railway Response Railway • 28 days ago


Railway
BOT

a month ago

There is no threshold that triggers a bulk reclaim. Continuous discard runs all the time and returns what it can, but the 16 KB block floor is structural: any block that is only partially freed stays allocated permanently. The gap between the reported metric and what df/du show inside the container will not close on its own regardless of how long the volume runs. Migrating to a fresh volume and deleting the old one is the only way to reset the footprint, and the growth will resume afterward as the workload writes and deletes data. For perspective, 1 GB of excess allocation costs roughly $0.15/month.


Status changed to Awaiting User Response Railway • 28 days ago


Railway

The `fstrim` failure ("Operation not permitted") is expected because containers do not have the privilege required for that operation. Volumes are mounted with continuous discard, so freed blocks are returned to the underlying storage layer over time automatically, but the reported usage settles above what `df`/`du` show inside the container due to a 16 KB block allocation floor (the storage allocates in 16 KB blocks while ext4 frees in 4 KB blocks, so partially-freed blocks cannot be reclaimed). The only way to fully reset a volume's footprint is to migrate the data to a fresh volume and delete the old one.

a month ago

I understand your reasoning that railway has the 16KB block floor.

But there must be way to re-organize the scattered data to free-up the partially freed space.

And again, swiping the disks every time is not really the solution which can be done again and again.


Status changed to Awaiting Railway Response Railway • 28 days ago


Railway
BOT

a month ago

There is no reorganization or defragmentation mechanism available inside the container, and fstrim is the only tool that could force a reclaim at the block level, which containers cannot run. The 16 KB allocation floor is a property of the storage layer itself, not something that can be compacted from within the filesystem. The gap between the reported metric and what df/du show will persist, and migrating to a fresh volume remains the only reset. For a Redis workload where the growth from 260 MB to 380 MB represents roughly $0.02/month in excess allocation cost, the practical impact is small.


Status changed to Awaiting User Response Railway • 28 days ago


a month ago

I still need help


Status changed to Awaiting Railway Response Railway • 28 days ago


Railway
BOT

a month ago

The position here has not changed: the reported volume usage settling above what df/du show is how the storage layer accounts for allocated blocks, and there is no mechanism to compact or reclaim that gap in place. Migrating data to a fresh volume and deleting the old one is the only reset, and the growth will resume as the workload writes and deletes data. For the range you described (260 MB to 380 MB), the excess allocation cost is a few cents per month.


Status changed to Awaiting User Response Railway • 28 days ago


Railway
BOT

20 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 20 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...