— Need resize2fs on Postgres volume (filesystem not expanded after resize)
cedarbluff-mark
PROOP

2 months ago

I cannot modify the Postgres service's start command because the patch is in a COMMITTED state (already deployed). The API won't allow changes to a committed patch.

This requires Railway engineering escalation. The block device /dev/zd64 is only accessible from within the Postgres container or from the host itself—not from other containers.

Escalation Details:

  • Service ID: 1a45d166-92dd-42bc-8f36-6e988283a169 (Postgres)
  • Command needed: resize2fs /dev/zd64
  • Context: Volume was resized from 1GB to 5GB in the Dashboard, but the ext4 filesystem wasn't expanded. Current filesystem: 879MB on a 5GB block device.
  • Why: The device is only visible inside the Postgres container. We need either:
    1. Railway engineering to run resize2fs /dev/zd64 directly on the host, OR
    2. The ability to modify the managed Postgres service's start command to include the resize step

Service ID: 1a45d166-92dd-42bc-8f36-6e988283a169

Project ID: 621a4bdf-bf95-41a3-ad1e-d3a3c5f840f1

Environment: production

The problem: I resized my Postgres volume from 1GB to 5GB through the Dashboard. The block device expanded

but the ext4 filesystem was not. The filesystem is 879MB on a 5GB block device. Postgres has been

crash-looping for hours.

The fix: Run resize2fs /dev/zd64 on the volume attached to this Postgres service. The device path was

confirmed by your Railway Agent during troubleshooting.

Current state: Postgres is online after we ran pg_resetwal to clear WAL files, but has only 8MB free and

is extremely fragile. The data (176k images, 9k listings, 22k verification records) is at risk.

This is a single command that would resolve the issue immediately. Can an engineer run it?

Solved

8 Replies

2 months ago

Your volume resize from 1 GB to 5 GB didn't fully apply on our end. We've corrected it and redeployed your Postgres service. You should now see approximately 4.6 GB of available disk space.

We've also shipped a fix to prevent this from happening on future volume resizes.


Status changed to Awaiting User Response Railway about 2 months ago


cedarbluff-mark
PROOP

2 months ago

Looks good. back online.

Thank you!


Status changed to Awaiting Railway Response Railway about 2 months ago


Status changed to Solved cedarbluff-mark about 2 months ago


cedarbluff-mark
PROOP

2 months ago

Same issue on the original volume. I had pulled a backup and you resized the backup, not the original, i directed you to the wrong one.

Can you run resize2fs on postgres-volume (vol_2a8m0noclie0keot) for project 621a4bdf? Same issue as the other volume you fixed yesterday — 879MB filesystem on a 5GB block device. Confirmed still crashing with "No space left on device" on pg_wal.


Status changed to Awaiting Railway Response Railway about 2 months ago


2 months ago

You would need to attach it to a service.


Status changed to Awaiting User Response Railway about 2 months ago


cedarbluff-mark
PROOP

2 months ago

I've attached postgres-volume (vol_2a8m0noclie0keot) to my Postgres-Qorj service.


Status changed to Awaiting Railway Response Railway about 2 months ago


cedarbluff-mark
PROOP

2 months ago

Can i get an update on this? system is down and has been for over 24 hours now.


brody

You would need to attach it to a service.

cedarbluff-mark
PROOP

2 months ago

trying to get some kind of confirmatino that you are aware this is ready for a fix and has been for 7 hours. our system has now been down for 24 hours.


2 months ago

Your volume resize from 1 GB to 5 GB didn't fully apply on our end. We've corrected it and redeployed your Postgres service. You should now see approximately 4.6 GB of available disk space.

We've also shipped a fix to prevent this from happening on future volume resizes.


Status changed to Awaiting User Response Railway about 2 months ago


Railway
BOT

2 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...