Volume usage metric still inflated after redeploy — requesting host-side block reclamation
zunoxi
HOBBYOP

a month ago

Project: donutkick

Volume: donutkick-volume

Volume ID: 43cbd771-0174-4408-98df-9ce58c937637

Service: donutkick

Environment: production

Issue:

Following up on the earlier finding that the Volume Usage metric

(GraphQL environment.volumeInstances.currentSizeMB) is reporting

~1.65 GB while actual filesystem usage is only ~132 MB (3% of 4.6 GB).

I redeployed the service multiple times today as suggested, to close

any lingering file descriptors and allow block reclamation. The

metric has not changed at all after several redeploys over several

hours.

Evidence (checked again just now):

  • df -h /data → 4.6G size, 132M used, 3% use
  • du -sh /data → 131M total
  • GraphQL query result: currentSizeMB: 1649.58, sizeMB: 5000
  • Multiple redeploys completed today, no change in reported metric

Context:

This volume was created a few weeks ago after migrating from an old

volume (new volume created, data copied, old volume deleted). We

suspect leftover block references from that migration and/or from

subsequent schema migrations that were never released back to ZFS.

Impact:

This is a billing metric, so we're currently being billed for ~1.65 GB

of storage usage despite ~132 MB of actual live data. Our own

usage-based alerting also reads this same inflated number.

Request:

Since redeploying did not resolve it, could you perform a host-side

ZFS block reclamation for this volume?

Solved

4 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


zunoxi
HOBBYOP

a month ago

Additional finding: railway volume list --json shows a second,

orphaned volume still present on this project:

  • Name: donutkick-volume-HZP8
  • Volume ID: 6321c0df-4f07-40e6-a769-bfae997109cc
  • currentSizeMB: 3787.8
  • isPendingDeletion: true
  • serviceName: null (detached)
  • deletedAt: 2026-07-20T01:00:49.763+00:00

This volume was deleted weeks ago as part of a migration to the

current volume (43cbd771-...), but it appears to never have been

fully purged — it's stuck in "pending deletion" and may still be

billed. Could you force-purge this volume as well?


echohack
EMPLOYEE

a month ago

The gap between your filesystem usage (132 MB) and the reported metric (1.65 GB) is a ZFS block-allocation discrepancy, not a data issue. The usage metric tracks allocated storage blocks rather than logical file size, and freed blocks are only returned when they are discarded back to the underlying storage layer. Running fstrim from inside the container is expected to fail (containers lack the required capability), so that is not a viable path.

Billing does track this metric, so the inflated number does affect your charges. The most reliable self-serve fix is to create a new volume, copy your ~132 MB of data to it, attach it to the service, and delete the current volume. A fresh volume starts with a clean block allocation that matches your actual data.

Regarding the second volume you found in pending deletion - when a volume with data is deleted, it enters a 48-hour safety retention window before it is permanently removed. This is a safeguard against accidental data loss and cannot be expedited. It clears on its own once the window passes, and you will not be billed for it after it is fully removed.


Status changed to Awaiting User Response Railway about 1 month ago


zunoxi
HOBBYOP

a month ago

Following up on my earlier thread (volume-usage-metric-still-inflated-after-0f7b9674). I migrated to a completely fresh volume on 2026-07-22 (00:39 KST). Real usage matched the metric immediately after migration. I'm not trying to guess at a root cause this time — just reporting the exact observed shape, since it doesn't match either of my two working theories.

Timeline:

2026-07-22 00:39 KST: migration completed, currentSizeMB ≈ real usage (~150-215MB).

2026-07-22 through 2026-07-23 17:00 KST (~40 hours): currentSizeMB climbed steadily up to ~3.7GB.

Zero deployments during this entire window.

Since 2026-07-23 17:00 KST through now: completely flat at ~3.7GB.

Real usage stayed flat at 215-243MB throughout — before, during, and after.

What doesn't add up: an hourly background job writes to SQLite every hour, all day, every day — if that drove the growth, it should still be climbing now, but it's been flat for over a day. A separate once-daily batch job should produce one big step per day, not a 40-hour continuous climb. Neither matches "climbs steadily then flatlines."

Questions:

What exactly does currentSizeMB measure, and how is it sampled?

Could a snapshot/point-in-time-recovery mechanism on your side pin and later release blocks independent of the container's own writes?

Is periodic volume migration really the only self-serve fix?

Does currentSizeMB directly drive billing, or is billing based on provisioned sizeMB (5GB) regardless?

Volume ID: ee8fb0c6-95d8-4946-826c-eac976b4bd10 / Project: donutkick


Status changed to Awaiting Railway Response Railway 28 days ago


a month ago

currentSizeMB reports the storage actually allocated to the volume, which can sit above the file sizes visible inside the container until freed space is reclaimed. Billing averages this figure over the billing period, so charges follow it rather than the provisioned 5 GB.

There are no snapshots on your volume, so no point-in-time mechanism on our side is pinning and releasing space independently of your writes.

Migration to a fresh volume and redeploys remain the self-serve options.


Status changed to Awaiting User Response Railway 28 days ago


Railway
BOT

20 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 20 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...