a month ago
Our production Postgres is stuck in a WAL recovery crash loop after running out of disk during a large insert.
The service never reaches a usable state. SELECT 1 through the public proxy fails with:
server closed the connection unexpectedly
The logs repeatedly show:
PANIC: could not write to file "pg_wal/xlogtemp.79": No space left on device
and then on restart:
FATAL: could not write to file "pg_wal/xlogtemp.30": No space left on device
Recovery reaches:
redo done at 0/A3FFFF98
then crashes again before accepting connections.
I tried to increase the volume size from the Railway dashboard but could not find a working path. I also created a new empty database service called Postgres Recovery as a fallback, but before rebuilding from scratch I need help recovering the existing production Postgres volume.
Could you please help with one of these:
- expand the existing Postgres volume/filesystem so WAL recovery can complete.
- mount/copy the existing volume into a larger replacement Postgres.
- restore the latest available internal backup/snapshot if one exists on Railway’s side.
Project: fabulous-renewal
Old service: Postgres
New fallback service: Postgres Recovery
Thank you in advance .
1 Replies
Status changed to Awaiting Railway Response Railway • 28 days ago
a month ago
Your Postgres volume is currently 500MB at ~96% capacity, which is why WAL recovery can't complete. You can resize it up to 5GB (the Hobby plan max) by clicking on the volume in your project canvas and using the Live Resize button in the volume settings. Since the volume is near full, the system will automatically perform an offline resize with data integrity checks, which restarts the service. After the resize completes, redeploy the Postgres service so WAL recovery has enough space to finish.
Status changed to Awaiting User Response Railway • 28 days ago
21 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 21 days ago