11 days ago
My Postgres volume is out of space and can't complete WAL recovery. I resized the volume to 100GB but it's still showing no space left on device. The volume mount appears to happen after Postgres tries to start. Service ID: da72c3c3-a6d4-4d0b-88eb-ca00b949fd3e
4 Replies
11 days ago
Seeing this in the error logs when I try to restart the service:
2026-02-28 05:59:46.437 UTC [30] FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device
11 days ago
IMPORTANT: PG_DUMP UR DB BEFORE ANY COMMANDS.
This is a known issue — resizing a Railway volume doesn't immediately resize the filesystem. The underlying block storage is expanded, but the ext4/xfs filesystem on it still reflects the old size until it's resized. Additionally, Postgres starts before the volume mount completes, so it sees a root filesystem with almost no space.
The core problem
FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device
Postgres needs to write WAL (write-ahead log) files to recover, but the disk is full so recovery fails, which means Postgres can't start, which means you can't clean anything up. It's a deadlock.
How to fix it
Step 1: Ask Railway support to resize the filesystem
Since you're on Pro, you have priority support. The volume resize to 100GB allocated the block storage, but the filesystem likely needs to be expanded by Railway's infrastructure team. Reply in your support thread or open a ticket with:
Service ID: da72c3c3-a6d4-4d0b-88eb-ca00b949fd3e Volume was resized from [old size] to 100GB but filesystem still shows the old capacity. Postgres can't start due to WAL recovery needing disk space. Please trigger a filesystem resize (
resize2fs) on the volume.
Step 2: If Railway can't resize the FS quickly — temporary workaround
Override the Postgres start command to clear space before Postgres launches:
Go to your Postgres service Settings > Deploy > Start Command
Set it to:
rm -f /var/lib/postgresql/data/pg_wal/*.tmp && rm -f /var/lib/postgresql/data/pg_wal/0000000100000000000000* && postgresWarning: Only delete
.tmpfiles safely. Deleting actual WAL segments (000000...) can cause data loss — only do this if you have backups or the data is recoverable. Start with just removing.tmpfiles:rm -f /var/lib/postgresql/data/pg_wal/*.tmp && postgresIf that frees enough space for recovery to complete, Postgres will start and you're good.
Step 3: If nothing works — nuclear option with backup
Spin up a new Postgres service with a 100GB volume
Once the old service starts (even briefly),
pg_dumpthe data outpg_restoreinto the new servicePoint your apps to the new service
Preventing this in the future
Set up disk usage alerts — Postgres WAL files can grow rapidly during heavy write periods
Configure
max_wal_sizeto limit WAL accumulation:ALTER SYSTEM SET max_wal_size = '2GB'; ALTER SYSTEM SET min_wal_size = '1GB';Run periodic
VACUUM FULLon large tables to reclaim spaceMonitor with:
SELECT pg_size_pretty(pg_database_size('your_db'));
Why this happened
Postgres WAL files grow when:
Heavy writes without checkpoints completing
Replication slots retaining old WAL segments
Long-running transactions preventing WAL cleanup
Check for stuck replication slots once you're back up:
SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
Drop any inactive slots that are retaining massive WAL.
TL;DR: Volume resize ≠ filesystem resize. You likely need Railway infra to run resize2fs on your volume. In the meantime, try clearing WAL temp files via a custom start command to free enough space for recovery.
11 days ago
Thanks, I thought that was the case. I guess I'll have to wait until Monday to get help with it
Status changed to Awaiting User Response Railway • 9 days ago
Status changed to Solved jake • 3 days ago