11 days ago
Project ID: 1bc27b20-ec6e-4480-9ea1-3722313a9fae
My PostgreSQL database is stuck in a crash loop. The pg_wal directory ran out of space. I expanded the volume but the database still cannot start with: could not write to file "pg_wal/xlogtemp.35": No space left on device
The data is intact but PostgreSQL cannot complete WAL recovery. I need someone to SSH into the container and clear old WAL files or run pg_resetwal so the database can start on the expanded volume.
This is a production outage. App is completely down.
3 Replies
11 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 11 days ago
11 days ago
Hey! You will need to clear the WAL files, first backup your volume and then set this as your start command:
bash -c 'rm -f /var/lib/postgresql/data/pgdata/postmaster.pid && runuser -u postgres -- pg_resetwal -f /var/lib/postgresql/data/pgdata && sleep 10 && exit 0'
If it writes "Write-ahead log reset" you're good to go, remove the start command and redeploy.
medim
Hey! You will need to clear the WAL files, first backup your volume and then set this as your start command: `bash -c 'rm -f /var/lib/postgresql/data/pgdata/postmaster.pid && runuser -u postgres -- pg_resetwal -f /var/lib/postgresql/data/pgdata && sleep 10 && exit 0'` If it writes "Write-ahead log reset" you're good to go, remove the start command and redeploy.
11 days ago
It is highly recommended against using pg_resetwal in this scenario. There is a high risk of data loss if OP goes with this suggestion.
The WAL contains all the recent transactions that have not yet been flushed to the database. Deleting it has a very high chance of corrupting the database.
Please don't recommend it to users, as it's only used as a last resort.
11 days ago
I'm aware that OP requested resetting WAL. But unless OP knows the possible consequences to this action; it's still better to advise against it.
Status changed to Closed medim • 11 days ago