Volume usage metric stuck at 4GB but actual usage is <1MB (ext4, df vs dashboard mismatch)
hiikun-bot
HOBBYOP

17 days ago

Hi, the volume usage metric for haruka-film-system-volume shows 83% (≈4GB / 5GB) used in the dashboard, but inside the running container the actual usage is only ~448K.

**Project:** HARUKA-FILME-SYSTEM

**Environment:** production

**Service:** haruka-film-system

**Volume:** haruka-film-system-volume

**Mount path:** /app/data

**Verification from inside the container railway ssh):**

$ df -h /app/data Filesystem Size Used Avail Use% Mounted on /dev/zd17168 4.6G 1.7M 4.5G 1% /app/data

$ du -sh /app/data 448K /app/data

$ mount | grep data /dev/zd17168 on /app/data type ext4 (rw,relatime,discard,stripe=8192)

$ find /app/data -type f | wc -l 4

$ lsof +L1 2>/dev/null (no output — no deleted-but-open files)


So the filesystem is ext4 (no snapshot retention semantics) and only ~448K is actually used, but the dashboard is reporting 4GB and showing a "High Volume Usage" warning.

Could you refresh / recompute the volume usage metric on your side? Happy to share the project ID via DM if needed.

Thanks!
$10 Bounty

1 Replies

Status changed to Awaiting Railway Response Railway 17 days ago


Status changed to Open Railway 16 days ago


dantor22
PROTop 10% Contributor

8 days ago

This is almost certainly a TRIM/discard mismatch between the filesystem and the underlying block device, not a bug in the

▎ metric itself.

▎ Quick context: on cloud volumes (Railway included) the usage metric is usually reported by the block storage layer, not by

▎ df inside the container. When you delete files, the filesystem marks blocks as free in its own bitmap, but unless a

▎ TRIM/discard is issued, the storage backend keeps thinking those blocks are still in use. Result: du/df say 448K, dashboard

▎ says 4GB.

▎ You mounted with discard which is correct — but that only emits discards for future deletes. Anything deleted before

▎ discard was active (or any large file written + deleted at some point) is still "dirty" from the block layer's perspective.

▎ Fix — run fstrim manually inside the container:

▎ fstrim -v /app/data

▎ Output will tell you how many bytes were trimmed. On a volume showing 4GB phantom usage you'll probably see something like

▎ 4 GiB trimmed. After that the dashboard metric should catch up (sometimes takes a few minutes to refresh).

▎ If fstrim isnt available in the image:

▎ apt-get update && apt-get install -y util-linux

▎ or run it from a temporary debian/alpine sidecar mounting the same volume.

▎ A couple gotchas worth checking too:

▎ - If you ever had a large file (logs, dumps, video render output) that got rotated/deleted, that fits the pattern

▎ perfectly.

▎ - lsof only catches files still open by a process — it wont show "deleted but never trimmed" blocks, so your lsof check was

▎ clean for a different reason.

▎ - ext4 also reserves ~5% for root by default + journal (~128MB), but thats nowhere near 4GB so its not that.

▎ Run the fstrim, post the output here and we'll know for sure. 99% its that.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...