2 months ago
I cannot modify the Postgres service's start command because the patch is in a COMMITTED state (already deployed). The API won't allow changes to a committed patch.
This requires Railway engineering escalation. The block device /dev/zd64 is only accessible from within the Postgres container or from the host itself—not from other containers.
Escalation Details:
- Service ID:
1a45d166-92dd-42bc-8f36-6e988283a169(Postgres) - Command needed:
resize2fs /dev/zd64 - Context: Volume was resized from 1GB to 5GB in the Dashboard, but the ext4 filesystem wasn't expanded. Current filesystem: 879MB on a 5GB block device.
- Why: The device is only visible inside the Postgres container. We need either:
- Railway engineering to run
resize2fs /dev/zd64directly on the host, OR - The ability to modify the managed Postgres service's start command to include the resize step
- Railway engineering to run
Service ID: 1a45d166-92dd-42bc-8f36-6e988283a169
Project ID: 621a4bdf-bf95-41a3-ad1e-d3a3c5f840f1
Environment: production
The problem: I resized my Postgres volume from 1GB to 5GB through the Dashboard. The block device expanded
but the ext4 filesystem was not. The filesystem is 879MB on a 5GB block device. Postgres has been
crash-looping for hours.
The fix: Run resize2fs /dev/zd64 on the volume attached to this Postgres service. The device path was
confirmed by your Railway Agent during troubleshooting.
Current state: Postgres is online after we ran pg_resetwal to clear WAL files, but has only 8MB free and
is extremely fragile. The data (176k images, 9k listings, 22k verification records) is at risk.
This is a single command that would resolve the issue immediately. Can an engineer run it?
8 Replies
2 months ago
Your volume resize from 1 GB to 5 GB didn't fully apply on our end. We've corrected it and redeployed your Postgres service. You should now see approximately 4.6 GB of available disk space.
We've also shipped a fix to prevent this from happening on future volume resizes.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
Looks good. back online.
Thank you!
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Solved cedarbluff-mark • about 2 months ago
2 months ago
Same issue on the original volume. I had pulled a backup and you resized the backup, not the original, i directed you to the wrong one.
Can you run resize2fs on postgres-volume (vol_2a8m0noclie0keot) for project 621a4bdf? Same issue as the other volume you fixed yesterday — 879MB filesystem on a 5GB block device. Confirmed still crashing with "No space left on device" on pg_wal.
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
I've attached postgres-volume (vol_2a8m0noclie0keot) to my Postgres-Qorj service.
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
Can i get an update on this? system is down and has been for over 24 hours now.
brody
You would need to attach it to a service.
2 months ago
trying to get some kind of confirmatino that you are aware this is ready for a fix and has been for 7 hours. our system has now been down for 24 hours.
2 months ago
Your volume resize from 1 GB to 5 GB didn't fully apply on our end. We've corrected it and redeployed your Postgres service. You should now see approximately 4.6 GB of available disk space.
We've also shipped a fix to prevent this from happening on future volume resizes.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 2 months ago
