Skip to content
English: Grafana volume usage keeps increasing: Railway reports 1.261 GB while guest filesystem usage is only ~53 MB
creativeson
OP

a month ago

Hello Railway team,

Please investigate the steadily increasing storage usage of our Grafana volume and the large discrepancy between Railway’s reported usage and the guest filesystem usage.

The service uses the official grafana/grafana-oss:12.4.3 Docker image, rather than a custom Grafana application. We have not established whether the cause is Grafana, its SQLite driver, storage allocation/reclamation, or platform accounting. We would appreciate host-side diagnostic evidence before attributing the issue to a particular component.

Please investigate the volume instance in mayhem, not production.

Observed behavior

The attached screenshot shows steadily increasing Volume Usage, with a reading of 1.14 GB on October 4, 2026 at 08:00 GMT+8.

Read-only checks performed on October 7, 2026 at approximately 17:09 GMT+8 produced these results:

Measurement Value

Railway DISK_USAGE_GB 1.2611584 GB

Guest filesystem usage reported by df 52,280 KiB, approximately 53.53 MB

du usage of /var/lib/grafana 51,084 KiB, approximately 52.31 MB

du usage of plugins 49,160 KiB, approximately 50.34 MB

Logical size of grafana.db 1,929,216 bytes, approximately 1.93 MB

Railway’s reported usage is approximately 24 times the guest filesystem usage, with a gap of approximately 1.21 GB.

Raw df output:

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/[DEVICE] 4725096 52280 4656432 1% /var/lib/grafana

Checks already performed

  • Grafana’s /api/health reports database: ok and version 12.4.3.
  • The png, pdf, and csv directories each occupy approximately 4 KiB. No substantial accumulation of export temporary files was observed.
  • /var/log/grafana occupies approximately 4 KiB. No substantial accumulation of file logs was observed.
  • No grafana.db-wal file was observed at sampling time.
  • The SQLite header’s read/write format bytes are both 1, consistent with rollback-journal format.
  • The ext4 mount already includes continuous discard:

/dev/[DEVICE] /var/lib/grafana ext4 rw,relatime,discard,stripe=8192

  • No deleted-but-open files under /var/lib/grafana were found among the readable /proc/*/fd entries. One entry was unreadable or disappeared during sampling, so this check does not conclusively rule out descriptor-related issues.
  • These checks did not delete files, run VACUUM or fstrim, restart/redeploy the service, or change its configuration.

Requested investigation

  1. Please inspect the host-side allocated blocks, metadata, journaling, discard reclamation, and accounting for this volume instance, and explain the approximately 1.21 GB discrepancy.
  2. Are there unreclaimed allocations or stale references? Can Railway reclaim unused space while preserving the Grafana data?
  3. If improperly closed file descriptors are identified as the cause, please provide the affected processes/files, retained space, and supporting diagnostic evidence so we can submit an actionable report to Grafana or the relevant driver upstream.
  4. How is the configured 5000 MB capacity enforced when guest free space remains substantial but platform allocated usage continues to grow? What condition would cause writes to fail?
  5. Does this discrepancy affect Volume billing? If it is an accounting issue, please review the charges for the current billing period.
  6. If a redeploy, maintenance operation, or fresh-volume migration is required, please explain its expected effect, downtime, data preservation and rollback procedure, and whether it addresses recurrence.

We found a similar SQLite case in which a Railway employee reported performing host-side cleanup to reclaim space:

https://station.railway.com/questions/railway-reports-volume-as-80x-larger-th-5e68de7c

We understand that its root cause may not apply to Grafana. Please investigate this instance using host-side evidence rather than assuming the same application-level cause.

Please do not delete data, backups, or volumes without our separate confirmatio

截圖 2026-10-07 下午5.50.12.png

n.

Solved

1 Reply

Railway
BOT

a month ago

The roughly 1.26 GB we report for grafana-volume in mayhem is the storage our side has allocated for the volume. It is not the size of the live files, so nothing is wrong inside your container and your Grafana data is fine. Space freed in ext4 only gets handed back when it is discarded. Our storage uses 16 KB blocks while ext4 frees space in 4 KB blocks, so a partly freed block can't be returned. That means this figure always sits above df and won't drop to match it.

Slow, steady growth like this while the guest filesystem stays flat is a known pattern on our storage layer. A redeploy (which remounts the volume) stops that background growth. It doesn't lower the figure already reported, and there's no fresh-volume migration or rebuild worth doing for this.

Leaked file descriptors aren't the cause here. Those show up when df reports more than du, and your df and du results match. We also don't run a manual host-side TRIM or reclaim on a volume.

Volume billing does use this allocated figure, at $0.15/GB-month averaged over the billing period. At about 1.2 GB above your live data, that comes to roughly $0.18 a month.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...