PostgreSQL stuck in crash-recovery loop - disk full, cannot complete recovery
senyil
PROOP

3 months ago

My PostgreSQL 15 database is completely stuck in a crash-recovery loop. The database ran out of disk space during a migration and now cannot complete recovery because it needs to write WAL files but has no disk space remaining.

What happened:

  1. I was upgrading Umami analytics from v2.9.0 to v3.0.3

  1. A migration tried to UPDATE ~2 million rows which filled the disk

  1. PostgreSQL crashed with "No space left on device"

  1. Now it cannot complete recovery because recovery itself needs disk space

Error from logs:

2025-12-14 12:27:38.836 UTC [29] FATAL:  could not write to file "pg_wal/xlogtemp.29": No space left on device

2025-12-14 12:27:38.843 UTC [2] LOG:  startup process (PID 29) exited with exit code 1

2025-12-14 12:27:38.843 UTC [2] LOG:  shutting down due to startup process failure

Recovery loop evidence:

The database keeps trying to recover but fails every ~30 seconds:

2025-12-14 12:25:53.472 UTC [29] LOG:  redo starts at 19/E40CB370

2025-12-14 12:25:59.830 UTC [29] LOG:  redo done at 1A/19FFE950 system usage: CPU: user: 2.91 s, system: 1.97 s, elapsed: 6.35 s

2025-12-14 12:25:59.863 UTC [29] FATAL:  could not write to file "pg_wal/xlogtemp.29": No space left on device

Important:

  • The redo completes successfully (redo done at 1A/19FFE950)

  • Recovery fails only when trying to write the checkpoint

  • Data appears intact - just needs disk space to complete recovery

Database size:

Approximately 3.7 GB based on earlier successful query

What I need:

  1. Increase the PostgreSQL volume size (I have Pro plan with 100GB allocation but can't find volume resize option)

  1. OR give me shell access to manually delete old WAL files to free space

Plan:

Railway Pro (100GB disk allocation available)

$10 Bounty

7 Replies

senyil
PROOP

3 months ago

SOLVED!


echohack
EMPLOYEE

3 months ago

Awesome! Glad you were able to fix it. Looks like your database now has space on its volume


Status changed to Awaiting User Response Railway 3 months ago


Status changed to Solved echohack 3 months ago


senyil
PROOP

2 months ago

Hey @echohack, I mistakenly increased the Volume Size to 100 GB, and now I can't use it for other projects. How can I downgrade it to 25 GB?


Status changed to Awaiting Railway Response Railway 2 months ago


echohack

Awesome! Glad you were able to fix it. Looks like your database now has space on its volume

senyil
PROOP

2 months ago

Hey @echohack, I mistakenly increased the Volume Size to 100 GB, and now I can't use it for other projects. How can I downgrade it to 25 GB?


2 months ago

We can't decrease the size of volumes, sorry


Status changed to Awaiting User Response Railway 2 months ago


ray-chen

We can't decrease the size of volumes, sorry

senyil
PROOP

2 months ago

Is this mean, if i create new Statamic project with postgres, this 100 GB will be shareable among? or do i need to migrate my umami to somewhere else to use railway again?


Status changed to Awaiting Railway Response Railway 2 months ago


2 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody 2 months ago


yusufmo1
PRO

2 months ago

Volumes are per-service, not shared across your account. Your Umami PostgreSQL has its own 100GB volume attached to it specifically. If you create a new project with a new PostgreSQL service, it gets its own separate volume. The 100GB on Umami doesn't block or consume anything from your other projects. you're good to create new databases elsewhere without migrating Umami away.


Loading...