Hey there! We've found the following might help you get unblocked faster:

• 🧵 Request to Increase Volume Limit to 10GB on Hobby Plan

• 🧵 Unable to create volume, got error: returned error 504 Gateway Timeout

• 🧵 Storage error in Railway

• 🧵 Do Railway Database and Storage Bucket share the same

If you find the answer from one of these, please let us know by solving the thread!

Hello,

The volume's usage is correct: 35GB of data, with the rest being filesystem metadata.

I'll open this up to the community so they can help you clear this out.

Best,
Brody

How can the database dump be 159 MB big and the production database (from which this staging db has its data from) be 0.95 GB big, but this staging db is 35GB ? And out of nowhere, it was sitting at the exact same storage level for weeks and suddenly it uses 38GB, even as it was in sleep at the time.

Can you SSH into the service and try using the df and du commands to see where the big files are? It's possible that something went wrong and a bunch of transaction logs or something got persisted to disk in the volume mount directory causing your issues. It would be a rare case, but I've seen that happen in non-railway environments that utilize docker volumes before.

Hey mykal, thanks for the suggestion, I think you are right, it looks like there is an issue with the WAL archiving looking at this numbers:

876M /var/lib/postgresql/data/pgdata/base

1.1G /var/lib/postgresql/data/pgdata/pg_wal

33G /var/lib/postgresql/data/pgdata/wal_archive

35G /var/lib/postgresql/data/pgdata

Note, it's the default railway postgres image and config.

Yeah seems like something is off with your WAL archival settings for sure.

Before doing anything, I would take a backup with pg_dump on your local machine just to make sure your data is safe.

After that you can try these from a database console:

Check your WAL settings:

SHOW archive_mode;
SHOW archive_command;
SHOW wal_level;

Set proper WAL archiving settings:

ALTER SYSTEM SET wal_keep_size = '1GB';
-- you can also set a custom wal_archive_command this way too; but do some resarch on what the best one would be

Disable WAL archiving altogether

if you're already doing regularly scheduled backups, you realistically won't need WAL setup at all

ALTER SYSTEM SET archive_mode = off;

If everything looks good, delete those old WAL files

just please make sure that your settings are a-okay and you have a backup that you can restore from if you have issues.

rm -rf /var/lib/postgresql/data/pgdata/wal_archive/*