2 months ago
Hi folks,
Firstly I would like to say I'm coming to stack architecture from a place of ignorance, which might be why I cannot understand a fix here.
I'm trying to restore a crashed volume to Postgres DB having used the 'live resize' version to increase the amount of space available. Restoring the volume and then attempting to mount it also seems to fail. Completely at a loss how to fix this. The railway AI assistant unfortunately seems to do more harm than damage and advised me the discarding volumes in the 'deploy' menu would not delete them, (which is what appear to have happened). The restore function of the backup seems to also take an extremely long time (2 hrs+). Here's Claude code's appraisal of the situation.
Here's exactly what's happening, step by step:
1. vol_o2vmd1a0pv0dwl0v is mounted — this is the original crashed volume, completely full at 5GB
2. PostgreSQL starts and replays the WAL (redo in progress, ~20 seconds) — this part succeeds
3. WAL replay finishes ("redo done") — the data itself is fine
4. PostgreSQL tries to write a checkpoint file to mark recovery as complete — this needs a tiny amount of disk space
5. FATAL: No space left on device — the volume is 100% full, can't write even a single byte
6. PostgreSQL shuts down, Railway restarts the container
7. Same volume vol_o2vmd1a0pv0dwl0v is mounted again — loop repeats forever
The core problem: Railway is ignoring your volume swap. Every restart keeps mounting the same full volume
vol_o2vmd1a0pv0dwl0v regardless of what you change in the UI.
You need to contact Railway support directly — tell them:
▎ "Volume vol_o2vmd1a0pv0dwl0v is full and stuck in a crash loop. I have a 20GB volume postgres-20gb that I cannot get
Railway to mount instead. The UI changes are not applying. Please detach vol_o2vmd1a0pv0dwl0v and attach my 20GB
volume."
2 Replies
2 months ago
Your Postgres service is in a crash loop because the volume is completely full. WAL recovery completes successfully but Postgres cannot write the checkpoint file needed to finish recovery. We can also see that a volume resize was attempted, but the service is still hitting "no space left" errors after the resize, which suggests the filesystem may not have extended properly. This requires manual intervention from our infrastructure team, and we've flagged it for them.
Status changed to Awaiting User Response Railway • 2 months ago
2 months ago
Following up on Chandrika's message: we've expanded the volume's filesystem to match the 20 GB you requested. The service has been redeployed and Postgres should now have enough space to complete WAL recovery and start up normally.
We've also shipped a fix to prevent this from happening on future volume resizes.
2 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 2 months ago