3 months ago
My PostgreSQL 15 database is completely stuck in a crash-recovery loop. The database ran out of disk space during a migration and now cannot complete recovery because it needs to write WAL files but has no disk space remaining.
What happened:
I was upgrading Umami analytics from v2.9.0 to v3.0.3
A migration tried to UPDATE ~2 million rows which filled the disk
PostgreSQL crashed with "No space left on device"
Now it cannot complete recovery because recovery itself needs disk space
Error from logs:
2025-12-14 12:27:38.836 UTC [29] FATAL: could not write to file "pg_wal/xlogtemp.29": No space left on device
2025-12-14 12:27:38.843 UTC [2] LOG: startup process (PID 29) exited with exit code 1
2025-12-14 12:27:38.843 UTC [2] LOG: shutting down due to startup process failure
Recovery loop evidence:
The database keeps trying to recover but fails every ~30 seconds:
2025-12-14 12:25:53.472 UTC [29] LOG: redo starts at 19/E40CB370
2025-12-14 12:25:59.830 UTC [29] LOG: redo done at 1A/19FFE950 system usage: CPU: user: 2.91 s, system: 1.97 s, elapsed: 6.35 s
2025-12-14 12:25:59.863 UTC [29] FATAL: could not write to file "pg_wal/xlogtemp.29": No space left on device
Important:
The redo completes successfully (redo done at 1A/19FFE950)
Recovery fails only when trying to write the checkpoint
Data appears intact - just needs disk space to complete recovery
Database size:
Approximately 3.7 GB based on earlier successful query
What I need:
Increase the PostgreSQL volume size (I have Pro plan with 100GB allocation but can't find volume resize option)
OR give me shell access to manually delete old WAL files to free space
Plan:
Railway Pro (100GB disk allocation available)
7 Replies
3 months ago
SOLVED!
3 months ago
Awesome! Glad you were able to fix it. Looks like your database now has space on its volume
Status changed to Awaiting User Response Railway • 3 months ago
Status changed to Solved echohack • 3 months ago
2 months ago
Hey @echohack, I mistakenly increased the Volume Size to 100 GB, and now I can't use it for other projects. How can I downgrade it to 25 GB?
Status changed to Awaiting Railway Response Railway • 2 months ago
echohack
Awesome! Glad you were able to fix it. Looks like your database now has space on its volume
2 months ago
Hey @echohack, I mistakenly increased the Volume Size to 100 GB, and now I can't use it for other projects. How can I downgrade it to 25 GB?
Status changed to Awaiting User Response Railway • 2 months ago
ray-chen
We can't decrease the size of volumes, sorry
2 months ago
Is this mean, if i create new Statamic project with postgres, this 100 GB will be shareable among? or do i need to migrate my umami to somewhere else to use railway again?
Status changed to Awaiting Railway Response Railway • 2 months ago
2 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open brody • 2 months ago
2 months ago
Volumes are per-service, not shared across your account. Your Umami PostgreSQL has its own 100GB volume attached to it specifically. If you create a new project with a new PostgreSQL service, it gets its own separate volume. The 100GB on Umami doesn't block or consume anything from your other projects. you're good to create new databases elsewhere without migrating Umami away.