3 months ago
Environment
- Service: PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2)
- Volume ID:
vol_qtkfr9mwjxjkrkl3 - Configured Volume Size: 40.00 GB
- Region: (see dashboard)
Problem Summary
The Railway volume is configured to 40GB, but the ext4 filesystem inside the volume was never expanded to fill the underlying block device.
As a result, PostgreSQL is reporting:
No space left on deviceeven though the volume appears to have sufficient capacity.
Filesystem Size (df -h)
Filesystem Size Used Avail Use% Mounted on
/dev/zd12208 434M 413M 12M 98% /var/lib/postgresqlThe ext4 filesystem is only 434MB, nearly full.
Block Device Size (lsblk)
zd12208 230:12208 0 37.3G 0 disk /var/lib/postgresqlThe underlying block device is actually 37.3GB, confirming that:
- The block device was expanded
- The ext4 filesystem was not resized to match
Mount Information
/dev/zd12208 on /var/lib/postgresql type ext4 (rw,relatime,discard,stripe=8192)Impact
- PostgreSQL is stuck in a crash loop
- WAL recovery requires writes but fails due to no space
- The database is currently down
- Recovery cannot be performed from inside the container
PostgreSQL Crash Log
2026-03-02 20:01:43.211 UTC [28] LOG: database system was interrupted while in recovery at 2026-03-02 20:01:36 UTC
2026-03-02 20:01:48.806 UTC [28] FATAL: could not write to file "pg_wal/xlogtemp.28": No space left on device
2026-03-02 20:01:48.816 UTC [2] LOG: shutting down due to startup process failureAttempted Resolution
Running resize2fs inside the container fails because:
- The block device node does not exist in
/dev mknodis not permitted due to container restrictions- The container does not have sufficient privileges to resize the filesystem
Requested Fix
Please run the following from the host level:
resize2fs /dev/zd12208Device details:
- Device name:
zd12208 - Major:Minor:
230:12208 - Expected size after resize: ~37.3GB
Once the ext4 filesystem is expanded to match the full block device size, PostgreSQL should be able to complete WAL recovery and start normally.
1 Replies
3 months ago
We can confirm this was a known issue where the ext4 filesystem was not resized to match the expanded block device. However, it looks like the original PostgreSQL service has since been deleted. Could you let us know what your current situation is? Specifically, whether you were able to recover your data or if you need assistance with the Postgres instance you currently have running.
Status changed to Awaiting User Response Railway • 3 months ago
2 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 2 months ago