Postgres stuck in crash recovery loop: No space left on device (pg_wal)
leonekkj
HOBBYOP

a month ago

My PostgreSQL service is stuck in a crash recovery boot loop. It cannot start because the volume is full and PostgreSQL cannot write the WAL temporary file needed for recovery.

Error from logs:

FATAL: could not write to file "pg_wal/xlogtemp.35": No space left on device

The database restarts, attempts crash recovery, reaches "redo done", then fails with the same error and shuts down again — indefinitely.

Root cause: The volume filled up during a VACUUM ANALYZE on the empresas table (6.8M rows, ~4.3 GB). The WAL generated by the vacuum consumed the remaining free space.

I have identified 3 unused indexes (~466 MB) ready to be dropped as soon as the server accepts connections. I just need the server to boot once.

Request: Could you temporarily expand the volume by 200–300 MB so the recovery can complete? Once the server is up I will immediately drop the unused indexes to bring usage well below the limit.

Solved

3 Replies

Railway
BOT

a month ago

Your Postgres volume is at 100% capacity and the service is crash-looping as confirmed in our logs. You can expand the volume yourself using Live Resize from the volume settings in your project. At 100% capacity, the system will perform an offline resize with data integrity checks, which restarts the service - this should allow crash recovery to complete. The Hobby plan supports volumes up to 5GB, so if your volume is already at that limit, you would need to upgrade to Pro (which supports up to 50GB) to resize further.


Status changed to Awaiting User Response Railway 26 days ago


Railway

Your Postgres volume is at 100% capacity and the service is crash-looping as confirmed in our logs. You can expand the volume yourself using [Live Resize](https://docs.railway.com/volumes#live-resizing-the-volume) from the volume settings in your project. At 100% capacity, the system will perform an offline resize with data integrity checks, which restarts the service - this should allow crash recovery to complete. The Hobby plan supports volumes up to 5GB, so if your volume is already at that limit, you would need to [upgrade to Pro](https://railway.com/account/billing) (which supports up to 50GB) to resize further.

leonekkj
HOBBYOP

a month ago

Could you temporarily expand the volume by 200–300 MB so the recovery can complete? Once the server is up I will immediately drop the unused indexes to bring usage well below the limit.


Status changed to Awaiting Railway Response Railway 25 days ago


Hey there,

Looking at your volume: you're at 4.99 GB used out of 5 GB, which is the Hobby plan cap. We can't temporarily expand beyond the plan limit from our side without you upgrading first.

The shortest path back to a booting database:

  1. Upgrade to Pro (gets you up to 50 GB volumes)
  2. Use Live Resize from your volume settings to add a small amount of headroom — even 1 GB extra will break the crash loop
  3. Once Postgres is up, drop those unused indexes you identified to clear well below the 5 GB mark
  4. If you want, downgrade back to Hobby once your usage is stable

Let me know if you hit any snags after the upgrade.

Thanks,

Angelo


Status changed to Awaiting User Response Railway 24 days ago


Railway
BOT

16 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 16 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...