a month ago
Project: donutkick
Volume: donutkick-volume
Volume ID: 43cbd771-0174-4408-98df-9ce58c937637
Service: donutkick
Environment: production
Issue:
Following up on the earlier finding that the Volume Usage metric
(GraphQL environment.volumeInstances.currentSizeMB) is reporting
~1.65 GB while actual filesystem usage is only ~132 MB (3% of 4.6 GB).
I redeployed the service multiple times today as suggested, to close
any lingering file descriptors and allow block reclamation. The
metric has not changed at all after several redeploys over several
hours.
Evidence (checked again just now):
df -h /data→ 4.6G size, 132M used, 3% usedu -sh /data→ 131M total- GraphQL query result: currentSizeMB: 1649.58, sizeMB: 5000
- Multiple redeploys completed today, no change in reported metric
Context:
This volume was created a few weeks ago after migrating from an old
volume (new volume created, data copied, old volume deleted). We
suspect leftover block references from that migration and/or from
subsequent schema migrations that were never released back to ZFS.
Impact:
This is a billing metric, so we're currently being billed for ~1.65 GB
of storage usage despite ~132 MB of actual live data. Our own
usage-based alerting also reads this same inflated number.
Request:
Since redeploying did not resolve it, could you perform a host-side
ZFS block reclamation for this volume?
4 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Additional finding: railway volume list --json shows a second,
orphaned volume still present on this project:
- Name: donutkick-volume-HZP8
- Volume ID: 6321c0df-4f07-40e6-a769-bfae997109cc
- currentSizeMB: 3787.8
- isPendingDeletion: true
- serviceName: null (detached)
- deletedAt: 2026-07-20T01:00:49.763+00:00
This volume was deleted weeks ago as part of a migration to the
current volume (43cbd771-...), but it appears to never have been
fully purged — it's stuck in "pending deletion" and may still be
billed. Could you force-purge this volume as well?
a month ago
The gap between your filesystem usage (132 MB) and the reported metric (1.65 GB) is a ZFS block-allocation discrepancy, not a data issue. The usage metric tracks allocated storage blocks rather than logical file size, and freed blocks are only returned when they are discarded back to the underlying storage layer. Running fstrim from inside the container is expected to fail (containers lack the required capability), so that is not a viable path.
Billing does track this metric, so the inflated number does affect your charges. The most reliable self-serve fix is to create a new volume, copy your ~132 MB of data to it, attach it to the service, and delete the current volume. A fresh volume starts with a clean block allocation that matches your actual data.
Regarding the second volume you found in pending deletion - when a volume with data is deleted, it enters a 48-hour safety retention window before it is permanently removed. This is a safeguard against accidental data loss and cannot be expedited. It clears on its own once the window passes, and you will not be billed for it after it is fully removed.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Following up on my earlier thread (volume-usage-metric-still-inflated-after-0f7b9674). I migrated to a completely fresh volume on 2026-07-22 (00:39 KST). Real usage matched the metric immediately after migration. I'm not trying to guess at a root cause this time — just reporting the exact observed shape, since it doesn't match either of my two working theories.
Timeline:
2026-07-22 00:39 KST: migration completed, currentSizeMB ≈ real usage (~150-215MB).
2026-07-22 through 2026-07-23 17:00 KST (~40 hours): currentSizeMB climbed steadily up to ~3.7GB.
Zero deployments during this entire window.
Since 2026-07-23 17:00 KST through now: completely flat at ~3.7GB.
Real usage stayed flat at 215-243MB throughout — before, during, and after.
What doesn't add up: an hourly background job writes to SQLite every hour, all day, every day — if that drove the growth, it should still be climbing now, but it's been flat for over a day. A separate once-daily batch job should produce one big step per day, not a 40-hour continuous climb. Neither matches "climbs steadily then flatlines."
Questions:
What exactly does currentSizeMB measure, and how is it sampled?
Could a snapshot/point-in-time-recovery mechanism on your side pin and later release blocks independent of the container's own writes?
Is periodic volume migration really the only self-serve fix?
Does currentSizeMB directly drive billing, or is billing based on provisioned sizeMB (5GB) regardless?
Volume ID: ee8fb0c6-95d8-4946-826c-eac976b4bd10 / Project: donutkick
Status changed to Awaiting Railway Response Railway • 28 days ago
a month ago
currentSizeMB reports the storage actually allocated to the volume, which can sit above the file sizes visible inside the container until freed space is reclaimed. Billing averages this figure over the billing period, so charges follow it rather than the provisioned 5 GB.
There are no snapshots on your volume, so no point-in-time mechanism on our side is pinning and releasing space independently of your writes.
Migration to a fresh volume and redeploys remain the self-serve options.
Status changed to Awaiting User Response Railway • 28 days ago
20 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 20 days ago
