PostgreSQL volume is full (No space left on device on pg_wal).
lluhk
PROOP

2 months ago

PostgreSQL volume is full (No space left on device on pg_wal). Volume size was increased to 100GB but the crash loop persists. Need shell access to the container or volume resize at the host level,

9 Replies

lluhk
PROOP

2 months ago

Details:

  • Volume ID: vol_xwn1v8gh3a06d3ti
  • Bind-mount ID: 873520e7-8657-4447-877d-98d70f18a81c
  • Exact error: FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device

The database keeps restarting every ~10 seconds and cannot accept any connections. I need shell access to the container or a volume resize at the host level to recover the database.


2 months ago

You can ssh into the container with Railway CLI


lluhk
PROOP

2 months ago

I tried railway ssh but got this error:

Failed to connect: Your application is not running or in a unexpected state

The container crashes every ~10 seconds so it never stays up long enough to accept SSH connections. What else can I do?


lluhk
PROOP

2 months ago

Update: I managed to connect via railway ssh and deleted old WAL files to free up space. That worked, but now the error changed to:

PANIC: could not locate a valid checkpoint record at 0/BDF7E00

Now the container crashes in less than 1 second, making it impossible to run any commands via railway ssh.

The data is intact (199MB in base/ directory). I just need to run pg_resetwal to fix the WAL, but the container is too fast to crash for me to do it.

Can you please run this command directly on the host?

runuser -u postgres -- pg_resetwal -f /var/lib/postgresql/data/pgdata

Volume ID: vol_obedizxdkadx473k


2 months ago

You can change the startup command to sleep 10000


2 months ago

This will stop postgres from running and should leave the files in an at-rest state for you to clean up


lluhk
PROOP

2 months ago

ok, thanks, I'll try to do this


lluhk
PROOP

2 months ago

It worked, thank you so much


2 months ago

You are most welcome!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...