Postgres Critical Failure: "No space left on device" after resizing 5GB (Full) to 250GB
kykkos
PROOP

16 days ago

Description of the issue: My Postgres service crashed because it reached the initial volume limit of 5GB (100% full). To resolve this, I resized the volume to 250GB. However, despite the resize, the service is now stuck in a crash loop. Even with 32GB RAM and 32 vCPUs allocated, Postgres fails during the recovery process.

Error message: The logs explicitly show: FATAL: could not write to file "pg_wal/xlogtemp.26": No space left on device.

Technical Details:

  • Service:Habyo / Postgres-Brb- / production

  • Image:ghcr.io/railwayapp-templates/postgres-ssl:16

  • Context: The volume was at 5GB/5GB (100%) before the resize. I am now at 250GB, but the system still behaves as if it's full.

  • Attempted fixes: 1. Increased RAM to 32GB to handle the heavy recovery process. 2. Attempted to clear /tmp and pg_stat_tmp via custom start command. 3. The redo process starts but fails at the same LSN every time due to lack of space.

Logs excerpt:

Plaintext

2026-02-24 17:51:00.652 UTC [26] LOG: database system was interrupted while in recovery at 2026-02-24 17:42:20 UTC
...
2026-02-24 17:51:43.757 UTC [26] FATAL: could not write to file "pg_wal/xlogtemp.26": No space left on device
2026-02-24 17:51:43.765 UTC [2] LOG: startup process (PID 26) exited with exit code 1

Request: It appears the filesystem resize was not properly propagated or the recovery process is blocked by the previous "Disk Full" state. Could you please check if the volume vol_evuuynq4gsvyltvq is healthy and correctly expanded to 250GB on your end?

Solved

3 Replies

sam-a
EMPLOYEE

16 days ago

We're looking into this now. It appears the volume resize to 250GB may not have fully propagated to the filesystem, which would explain why Postgres still sees the disk as full during WAL recovery. We've escalated this to our platform team to investigate the volume (vol_evuuynq4gsvyltvq) and ensure the resize is applied at the filesystem level.


Status changed to Awaiting User Response Railway 16 days ago


Railway
BOT

16 days ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!


sam-a

We're looking into this now. It appears the volume resize to 250GB may not have fully propagated to the filesystem, which would explain why Postgres still sees the disk as full during WAL recovery. We've escalated this to our platform team to investigate the volume (vol_evuuynq4gsvyltvq) and ensure the resize is applied at the filesystem level.

kykkos
PROOP

16 days ago

It finally works! Thank you so much, I've been trying to solve this problem for hours.


Status changed to Awaiting Railway Response Railway 16 days ago


Status changed to Solved kykkos 16 days ago


Loading...