The dashboard's "Volume Usage" metric measures the ZFS zvol's allocated blocks, not the logical size of live data inside the container's filesystem. When files are deleted or rewritten, freed blocks are only returned to ZFS after they are discarded/TRIMmed. Volumes are mounted with continuous discard, so this normally reconciles on its own over minutes to hours, and a lag after a resize or heavy writes is expected. Billing is based on this same metric (averaged over the billing period), not on what df/du report inside the container. For Postgres specifically, plain VACUUM marks pages reusable within Postgres but does not return space to the filesystem. Only VACUUM FULL or pg_repack rewrites tables compactly and actually shrinks the on-disk files. Running fstrim inside the container will fail ("Operation not permitted") because containers lack CAP_SYS_ADMIN, and that is expected. If the metric stays elevated after a VACUUM FULL or pg_repack and continuous discard has had time to catch up, we can run a host-side reclaim to force any remaining blocks to be returned. No customer-side action is needed for the discard layer itself, only for Postgres-level bloat reclamation via VACUUM FULL or pg_repack.

Thank you, but this does not resolve the issue. I still need help from a Railway engineer.

The recommendation to run VACUUM FULL does not match the measured data:

• The active database is only 37 MB in total.
• All databases combined are approximately 59 MB.
• WAL is 112 MB.
• df reports 172 MB used.
• Default du also reports 172 MB of allocated filesystem blocks, not merely apparent file size.
• The two largest application tables are only 14 MB each.
• They have approximately 39,000 rows and only about 500 estimated dead rows each.

Therefore, even completely rewriting a 37 MB database cannot account for or reclaim the unexplained approximately 527 MB difference between the 699 MB Railway metric and the 172 MB guest-filesystem allocation.

Running VACUUM FULL would require production locks, create additional writes and WAL, and cannot reclaim hundreds of megabytes from database files that are only 37 MB in total.

Please escalate this to a human Railway engineer for host-side inspection.

Specifically, please determine:

1. Whether the 527 MB difference consists of ZFS blocks awaiting DISCARD/TRIM.
2. Whether Railway can initiate host-side reclamation, since customer-side fstrim is unavailable.
3. Whether the dashboard value is expected to reconcile automatically and within what timeframe.
4. Whether billing continues to include these blocks while the customer has no ability to reclaim them.
5. Why VACUUM FULL would be relevant when the entire active database is only 37 MB and df/du already report the blocks as free.

I do not authorize or intend to run VACUUM FULL, pg_repack, restart, or redeploy without evidence that the issue exists within PostgreSQL files.

Your analysis is correct: with df/du already at 172 MB, the gap is entirely at the ZFS DISCARD layer, not within Postgres, and VACUUM FULL would not address it. The volume metric has already dropped from 699 MB to approximately 431 MB since you filed this thread, confirming continuous discard is actively reconciling.

Thank you for confirming this. I will not run VACUUM FULL, pg_repack, restart, or redeploy, and I will allow continuous discard to finish reconciling.

One question remains: is volume billing calculated from the temporarily elevated ZFS Volume Usage metric during the DISCARD delay, or is this transient unreclaimed allocation excluded or adjusted?

Also, is there an approximate timeframe after which I should report back if the dashboard metric does not converge toward the 172 MB reported by df/du?

Billing is based on the dashboard metric (ZFS allocated blocks), averaged over the billing period, so the transient elevation does affect the average until discard catches up. As reconciliation progresses (it has already dropped from 699 MB to ~431 MB), the average falls accordingly. If the metric has not converged toward your 172 MB df/du figure within about 48 hours, reply here so the situation can be reassessed.

More than 48 hours have now passed, but the dashboard metric has not converged toward the guest-filesystem usage.

Instead, Railway Volume Usage increased to 861.27 MB as of July 24 at 15:25 GMT+2. This is higher than the previous values of 699 MB and approximately 431 MB.

I rechecked the mounted filesystem at July 24, 15:56 GMT+2:

Filesystem      Size  Used  Avail  Use%  Mounted on
/dev/zd6464     4.4G  174M  4.2G   4%   /var/lib/postgresql/data

du reported:

16K   /var/lib/postgresql/data/lost+found
40K   /var/lib/postgresql/data/certs
174M  /var/lib/postgresql/data
174M  /var/lib/postgresql/data/pgdata

The guest filesystem increased by only approximately 2 MB, from 172 MB to 174 MB, while the Railway ZFS metric increased to 861.27 MB. The unexplained difference is now approximately 687 MB.

The metric is increasing rather than reconciling within the 48-hour timeframe you provided.

Please reassess and escalate this for host-side inspection and ZFS block reclamation. Customer-side DISCARD/fstrim is unavailable, and the PostgreSQL filesystem already reports the blocks as free.

I have not run VACUUM FULL, pg_repack, restart, redeploy, or any other destructive production maintenance.

The volume metric is now at ~864 MB with df/du still at 174 MB, so continuous discard has not reconciled as expected. The most common cause of this pattern is leaked file descriptors - a process holding deleted files open prevents the filesystem from releasing those blocks to ZFS. A redeploy closes all open handles and lets reclamation proceed. Since you have not redeployed since the resize, that is the recommended next step. It does not touch your data (the volume is persistent), and no VACUUM FULL or pg_repack is needed given your database is only 37 MB.

I checked for deleted files held open inside the PostgreSQL container:

find /proc/[0-9]*/fd -lname '* (deleted)' -ls 2>/dev/null

The command returned no output.

Additionally, both df and du report exactly 174 MB. A deleted file held open would normally remain included in df while disappearing from du, but that discrepancy is not present here.

Please confirm whether you still recommend redeploying PostgreSQL and whether the intended effect is to force a volume remount or host-side ZFS reclamation rather than close a confirmed leaked file descriptor.

Before proceeding, please also confirm:

1. Only the PostgreSQL service should be redeployed.
2. The persistent volume and its data will remain attached and unchanged.
3. The expected database downtime.
4. Whether a fresh Railway backup should be created immediately beforehand.

Your file descriptor check is thorough and rules out that hypothesis. A redeploy is still worth trying because the unmount/remount cycle can trigger ZFS-level block reclamation independent of leaked handles. To your questions: yes, redeploy only the PostgreSQL service. Your persistent volume and its data remain attached and unchanged through a redeploy. Downtime is typically under a minute. Taking a volume backup beforehand from the service's Backups tab is a reasonable precaution.

The PostgreSQL-only redeploy has been completed successfully.

The persistent volume remained attached, both services are Online, and the new container reports:

Filesystem      Size  Used  Avail  Use%  Mounted on
/dev/zd6464     4.4G  159M  4.2G   4%   /var/lib/postgresql/data

du also reports 159 MB.

The dashboard metric was still approximately 1.22 GB immediately after redeploy, although the right edge of the graph appears to have started dropping. I will monitor it for the next several hours and report back if it does not continue converging toward the 159 MB guest-filesystem value.

An update after digging into your volume's storage accounting. Two corrections to what we told you earlier.

First, the metric stopped growing after your redeploy. It's been flat at about 1.05 GB for two days now, so nothing is still leaking.

Second, it won't converge to the 159 MB that df reports, and waiting won't change that. Two reasons. The resize automatically created a safety backup named "Online resize to 5000MB". It's still on the volume, it holds on to blocks your filesystem has since freed, and it's what shows as Backup Usage. You can delete it from the volume's Backups tab, which releases that space. Separately, the underlying storage allocates in 16 KB blocks while your filesystem works in 4 KB blocks, so partially freed blocks can't always be returned. That keeps allocation somewhat above df permanently.

On billing: you're billed on this metric plus backup usage at $0.15 per GB month, averaged over the cycle. Your entire current gap works out to roughly $0.20 per month.

If the numbers still look off to you after deleting that backup, reply here and we'll take another look.

Railway Team

Thank you for the detailed investigation and clarification.

This resolves the issue for me. I understand now that:

• the metric is no longer growing,
• the remaining gap is explained by the Online resize to 5000MB safety backup,
• and the rest is due to underlying storage block allocation overhead.

I will keep the resize backup for now, since it is currently my only recovery point, and I understand that this means the reported usage will remain above the guest filesystem usage.

Thanks again for the help. I’m considering this resolved and will close the thread.