a month ago
I’m trying to understand a large mismatch between Railway’s volume metric and the mounted filesystem.
My volume originally had a 500 MB cap and Railway usage climbed into roughly the 400–450 MB range while the actual filesystem was only about 186 MB used.
I resized the volume to 1 GB, and Railway’s reported usage then rose quickly:
- ~443 MB yesterday morning
- ~470 MB an hour later
- ~506 MB by early afternoon
- ~748 MB today
But inside the running container:
- df shows only about 194 MB used
- about 717 MB free/app-available
- visible files total about 194.2 MB, which accounts for essentially all filesystem usage
So Railway’s metric increased by roughly 300 MB without corresponding file growth.
What I’m trying to understand is:
- What exactly is Railway counting that explains the missing ~554 MB between the dashboard metric and df?
- Why would that provider-reported usage continue climbing rapidly while the filesystem stays essentially flat?
- Could the 500 MB → 1 GB resize have exposed or changed underlying block allocation behavior?
- Is this expected with create/delete churn, retained allocated blocks, snapshots, thin provisioning, or something else?
- Can that provider-side usage be reclaimed, or is a fresh volume the only reset?
- If the Railway metric reaches 100% while df still has free space, will writes actually fail?
I found another Railway thread where the configured volume size and the actual mounted filesystem diverged and Railway staff had to extend the filesystem separately, so I’m trying to understand how the dashboard metric, storage-layer allocation, and guest filesystem relate to one another.
I’m mainly looking for a technical explanation of why this metric is growing this way, not just generic advice to delete files.
4 Replies
a month ago
The volume metric counts the blocks allocated on the storage layer underneath your filesystem, not the live data df reports. Files deleted or rewritten inside the container only hand their blocks back to the storage layer once they are discarded, which happens on its own over minutes to hours. The storage layer also allocates in 16 KB blocks while the container's filesystem frees space in 4 KB blocks, so a partly freed 16 KB block can't be returned. With many small files or frequent rewrites, that can keep allocation at up to roughly 4x what df shows, permanently.
The jump after resizing to 1 GB is expected. Initializing filesystem metadata for the new capacity allocates real blocks, which can add hundreds of MB. After that the figure can keep climbing a few MB per hour for days while your files stay flat. None of that is your data, and nothing is wrong inside your container. A redeploy, which remounts the volume, stops this background growth. Snapshots are not part of this figure.
This allocation can't be reclaimed manually. Running fstrim inside the container returns "Operation not permitted" because containers don't have that privilege, and the figure will settle somewhere above df, not at it. Billing does use this metric at $0.15/GB-month, so your roughly 0.55 GB gap works out to about $0.08 per month.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
The volume metric counts the blocks allocated on the storage layer underneath your filesystem, not the live data `df` reports. Files deleted or rewritten inside the container only hand their blocks back to the storage layer once they are discarded, which happens on its own over minutes to hours. The storage layer also allocates in 16 KB blocks while the container's filesystem frees space in 4 KB blocks, so a partly freed 16 KB block can't be returned. With many small files or frequent rewrites, that can keep allocation at up to roughly 4x what `df` shows, permanently. The jump after resizing to 1 GB is expected. Initializing filesystem metadata for the new capacity allocates real blocks, which can add hundreds of MB. After that the figure can keep climbing a few MB per hour for days while your files stay flat. None of that is your data, and nothing is wrong inside your container. A redeploy, which remounts the volume, stops this background growth. Snapshots are not part of this figure. This allocation can't be reclaimed manually. Running `fstrim` inside the container returns "Operation not permitted" because containers don't have that privilege, and the figure will settle somewhere above `df`, not at it. Billing does use this metric at $0.15/GB-month, so your roughly 0.55 GB gap works out to about $0.08 per month.
a month ago
Thanks — this explains the mismatch much better. Before I make any production changes, can you clarify a few operational points?
- When you say a redeploy/remount stops the background growth, does it merely stop further post-resize allocation, or does it also reduce/reclaim any of the already-reported ~748 MB?
- If Railway’s storage-layer metric reaches 100% while df still shows substantial free space, what actually happens? Can application writes fail with ENOSPC, or is df free space still usable?
- Is the 1 GB configured volume size enforced against the storage-layer allocated-block metric, the guest filesystem usage shown by df, or some combination of the two?
- After a resize, roughly how long can this metadata/background allocation continue before it normally settles? Is “a few MB/hour for days” expected even with almost no application writes?
- Is there any way to distinguish post-resize metadata/background allocation from allocation caused by my application’s normal create/delete/rewrite churn?
- If the allocated-block metric remains high after redeploy but stops climbing, is that considered normal/safe indefinitely?
- If I move temporary high-churn workspaces off the persistent volume and onto the service’s ephemeral filesystem, should that materially reduce future growth in this metric?
- Does migrating the same live files to a fresh volume reset this allocated-block high-water usage close to actual filesystem usage, even though it may accumulate again over time?
- Are there any Railway-side alerts or metrics I should monitor in addition to df, ENOSPC, and write failures to know whether the volume is actually approaching an operational limit?
My main goal is to know whether this is just an accounting/high-water effect I can safely monitor, or whether it can still cause a production write failure despite df showing free space.
Status changed to Awaiting Railway Response Railway • about 1 month ago
Railway
The volume metric counts the blocks allocated on the storage layer underneath your filesystem, not the live data `df` reports. Files deleted or rewritten inside the container only hand their blocks back to the storage layer once they are discarded, which happens on its own over minutes to hours. The storage layer also allocates in 16 KB blocks while the container's filesystem frees space in 4 KB blocks, so a partly freed 16 KB block can't be returned. With many small files or frequent rewrites, that can keep allocation at up to roughly 4x what `df` shows, permanently. The jump after resizing to 1 GB is expected. Initializing filesystem metadata for the new capacity allocates real blocks, which can add hundreds of MB. After that the figure can keep climbing a few MB per hour for days while your files stay flat. None of that is your data, and nothing is wrong inside your container. A redeploy, which remounts the volume, stops this background growth. Snapshots are not part of this figure. This allocation can't be reclaimed manually. Running `fstrim` inside the container returns "Operation not permitted" because containers don't have that privilege, and the figure will settle somewhere above `df`, not at it. Billing does use this metric at $0.15/GB-month, so your roughly 0.55 GB gap works out to about $0.08 per month.
a month ago
Thanks — the explanation about allocated storage-layer blocks versus df makes sense. I haven’t gotten a follow-up reply yet, so I’m trying to clarify the remaining operational questions before I change anything in production.
- If Railway’s reported volume usage reaches 100% while df still shows substantial free space, can writes actually fail with ENOSPC, or does the guest filesystem free space remain usable?
- Is the configured volume limit enforced against the provider allocation metric, the guest filesystem, or both?
- You mentioned that a redeploy/remount stops the post-resize background growth. Does it only stop further growth, or can it also reduce/reclaim already allocated usage?
- If the Railway metric stays high after redeploy but df remains healthy and writes succeed, is that considered normal/safe?
- For a workload with lots of temporary create/delete activity, would moving scratch work off the persistent volume reduce future provider-side allocation growth?
I’m mainly trying to determine whether this is just an accounting/high-water effect or whether the provider metric itself can eventually become a real production write limit.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This is an accounting effect, not a write limit. The configured 1 GB is enforced only as the size of the block device your filesystem sits on, so the filesystem inside the container is what decides when writes fail. If the dashboard figure reaches 100% while df still shows free space, that free space stays usable. ENOSPC happens when df runs out, not when the dashboard figure does.
A redeploy only stops the post-resize background growth. It does not bring the existing figure down. Blocks your files free up are returned on their own over minutes to hours, but the 16 KB block floor means the figure settles above df, not at it.
A figure that stays high and flat while df is healthy and writes succeed is normal, and it is safe to leave as is. It does affect billing, at $0.15/GB-month.
The gap comes from small-file create/delete/rewrite churn on the volume. Moving temporary scratch work onto the service's ephemeral filesystem takes that churn off the volume, so it should reduce future growth in this figure. Anything stored there is lost on every redeploy.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago