Postgres out of space
twillsyndhir
PROOP

11 days ago

My Postgres volume is out of space and can't complete WAL recovery. I resized the volume to 100GB but it's still showing no space left on device. The volume mount appears to happen after Postgres tries to start. Service ID: da72c3c3-a6d4-4d0b-88eb-ca00b949fd3e

Solved

4 Replies

twillsyndhir
PROOP

11 days ago

Seeing this in the error logs when I try to restart the service:

2026-02-28 05:59:46.437 UTC [30] FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device


rafchik-dipstick
HOBBY

11 days ago

IMPORTANT: PG_DUMP UR DB BEFORE ANY COMMANDS.

This is a known issue — resizing a Railway volume doesn't immediately resize the filesystem. The underlying block storage is expanded, but the ext4/xfs filesystem on it still reflects the old size until it's resized. Additionally, Postgres starts before the volume mount completes, so it sees a root filesystem with almost no space.

The core problem

FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device

Postgres needs to write WAL (write-ahead log) files to recover, but the disk is full so recovery fails, which means Postgres can't start, which means you can't clean anything up. It's a deadlock.

How to fix it

Step 1: Ask Railway support to resize the filesystem

Since you're on Pro, you have priority support. The volume resize to 100GB allocated the block storage, but the filesystem likely needs to be expanded by Railway's infrastructure team. Reply in your support thread or open a ticket with:

Service ID: da72c3c3-a6d4-4d0b-88eb-ca00b949fd3e Volume was resized from [old size] to 100GB but filesystem still shows the old capacity. Postgres can't start due to WAL recovery needing disk space. Please trigger a filesystem resize (resize2fs) on the volume.

Step 2: If Railway can't resize the FS quickly — temporary workaround

Override the Postgres start command to clear space before Postgres launches:

  1. Go to your Postgres service Settings > Deploy > Start Command

  2. Set it to:

    rm -f /var/lib/postgresql/data/pg_wal/*.tmp && rm -f /var/lib/postgresql/data/pg_wal/0000000100000000000000* && postgres
    

    Warning: Only delete .tmp files safely. Deleting actual WAL segments (000000...) can cause data loss — only do this if you have backups or the data is recoverable. Start with just removing .tmp files:

    rm -f /var/lib/postgresql/data/pg_wal/*.tmp && postgres
    
  3. If that frees enough space for recovery to complete, Postgres will start and you're good.

Step 3: If nothing works — nuclear option with backup

  1. Spin up a new Postgres service with a 100GB volume

  2. Once the old service starts (even briefly), pg_dump the data out

  3. pg_restore into the new service

  4. Point your apps to the new service

Preventing this in the future

  • Set up disk usage alerts — Postgres WAL files can grow rapidly during heavy write periods

  • Configure max_wal_size to limit WAL accumulation:

    ALTER SYSTEM SET max_wal_size = '2GB';
    ALTER SYSTEM SET min_wal_size = '1GB';
    
  • Run periodic VACUUM FULL on large tables to reclaim space

  • Monitor with: SELECT pg_size_pretty(pg_database_size('your_db'));

Why this happened

Postgres WAL files grow when:

  • Heavy writes without checkpoints completing

  • Replication slots retaining old WAL segments

  • Long-running transactions preventing WAL cleanup

Check for stuck replication slots once you're back up:

SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;

Drop any inactive slots that are retaining massive WAL.

TL;DR: Volume resize ≠ filesystem resize. You likely need Railway infra to run resize2fs on your volume. In the meantime, try clearing WAL temp files via a custom start command to free enough space for recovery.


twillsyndhir
PROOP

11 days ago

Thanks, I thought that was the case. I guess I'll have to wait until Monday to get help with it


Hey, you should be good now - can you try again?


Status changed to Awaiting User Response Railway 9 days ago


Status changed to Solved jake 3 days ago


Loading...