Volume filesystem not resized to match block device (ext4 only 434MB on 37.3GB block device)
ventali
PROOP

3 months ago

Environment

  • Service: PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2)
  • Volume ID:vol_qtkfr9mwjxjkrkl3
  • Configured Volume Size: 40.00 GB
  • Region: (see dashboard)

Problem Summary

The Railway volume is configured to 40GB, but the ext4 filesystem inside the volume was never expanded to fill the underlying block device.

As a result, PostgreSQL is reporting:

No space left on device

even though the volume appears to have sufficient capacity.

Filesystem Size (df -h)

Filesystem      Size  Used Avail Use% Mounted on
/dev/zd12208    434M  413M   12M  98% /var/lib/postgresql

The ext4 filesystem is only 434MB, nearly full.

Block Device Size (lsblk)

zd12208 230:12208  0  37.3G  0 disk /var/lib/postgresql

The underlying block device is actually 37.3GB, confirming that:

  • The block device was expanded
  • The ext4 filesystem was not resized to match

Mount Information

/dev/zd12208 on /var/lib/postgresql type ext4 (rw,relatime,discard,stripe=8192)

Impact

  • PostgreSQL is stuck in a crash loop
  • WAL recovery requires writes but fails due to no space
  • The database is currently down
  • Recovery cannot be performed from inside the container

PostgreSQL Crash Log

2026-03-02 20:01:43.211 UTC [28] LOG:  database system was interrupted while in recovery at 2026-03-02 20:01:36 UTC
2026-03-02 20:01:48.806 UTC [28] FATAL:  could not write to file "pg_wal/xlogtemp.28": No space left on device
2026-03-02 20:01:48.816 UTC [2] LOG:  shutting down due to startup process failure

Attempted Resolution

Running resize2fs inside the container fails because:

  • The block device node does not exist in /dev
  • mknod is not permitted due to container restrictions
  • The container does not have sufficient privileges to resize the filesystem

Requested Fix

Please run the following from the host level:

resize2fs /dev/zd12208

Device details:

  • Device name: zd12208
  • Major:Minor: 230:12208
  • Expected size after resize: ~37.3GB

Once the ext4 filesystem is expanded to match the full block device size, PostgreSQL should be able to complete WAL recovery and start normally.

Solved

1 Replies

sam-a
EMPLOYEE

3 months ago

We can confirm this was a known issue where the ext4 filesystem was not resized to match the expanded block device. However, it looks like the original PostgreSQL service has since been deleted. Could you let us know what your current situation is? Specifically, whether you were able to recover your data or if you need assistance with the Postgres instance you currently have running.


Status changed to Awaiting User Response Railway 3 months ago


Railway
BOT

2 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...