2 months ago
PostgreSQL volume is full (No space left on device on pg_wal). Volume size was increased to 100GB but the crash loop persists. Need shell access to the container or volume resize at the host level,
9 Replies
Details:
- Volume ID: vol_xwn1v8gh3a06d3ti
- Bind-mount ID: 873520e7-8657-4447-877d-98d70f18a81c
- Exact error: FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device
The database keeps restarting every ~10 seconds and cannot accept any connections. I need shell access to the container or a volume resize at the host level to recover the database.
2 months ago
You can ssh into the container with Railway CLI
I tried railway ssh but got this error:
Failed to connect: Your application is not running or in a unexpected state
The container crashes every ~10 seconds so it never stays up long enough to accept SSH connections. What else can I do?
Update: I managed to connect via railway ssh and deleted old WAL files to free up space. That worked, but now the error changed to:
PANIC: could not locate a valid checkpoint record at 0/BDF7E00
Now the container crashes in less than 1 second, making it impossible to run any commands via railway ssh.
The data is intact (199MB in base/ directory). I just need to run pg_resetwal to fix the WAL, but the container is too fast to crash for me to do it.
Can you please run this command directly on the host?
runuser -u postgres -- pg_resetwal -f /var/lib/postgresql/data/pgdata
Volume ID: vol_obedizxdkadx473k
2 months ago
You can change the startup command to sleep 10000
2 months ago
This will stop postgres from running and should leave the files in an at-rest state for you to clean up
2 months ago
You are most welcome!