Skip to content
PITR base backups fill page cache that's billed as memory: is a fix still planned?
58f-ops
OP

8 hours ago

Hi Railway team,

The subject of cache has come up twice already as far as I can tell, on these previous threads:

https://station.railway.com/questions/railway-postgres-pitr-backup-leaves-6-gb-e762df27

https://station.railway.com/questions/postgres-memory-stays-high-after-restart-ea405100

Both are marked solved, but the root cause is still there so I'm trying my luck here.

The setup: postgres-ssl:18 with managed PITR, about 21 GB of data, default limit (32 GB). Over the last 30 days, billed memory averaged 12.3 GB and peaked at 28 GB, with a spike after every base backup. A memory.stat reading taken this morning:

anon 0.25 GB

shmem 2.26 GB (shared_buffers)

kernel 0.26 GB

active_file 0.08 GB

inactive_file 6.77 GB (read once, by the backup)

Postgres is really using about 2.9 GB. A manual dd iflag=nocache count=0 over $PGDATA brought billed memory down from 9.6 to 2.8 GB right away, so this is all clean, read-once cache.

About the usually suggested workarounds:

  • **Memory limit. **It's the only option that keeps managed PITR, but it's a bet. A hard memory.max has to sit above Postgres's peak non-reclaimable memory: shared_buffers, work_mem x concurrent sorts and hashes, maintenance_work_mem during VACUUM or CREATE INDEX, and so on... That peak is hard to predict, and guessing too low = OOM kill and crash recovery on a production database.

Even when the limit holds, the cache refills up to it, so the bill settles at the limit, not at real usage. In practice we can only cut the bill by going as close to the danger zone as we dare.

  • **Custom image **(nocache around the backup command). We built and tested this in an isolated project. Page-cache residency on the test data stayed low, and the restore checked out. But managed PITR refuses to enable on a Git-built service.

  • Custom start command with a cache-trim loop. This is also incompatible with PITR/HA, as far as we can tell.

So every fix that targets the cause loses managed PITR, and the one that keeps it trades the bill against an OOM risk.

What would solve it on your side, any one of these:

  1. In the image: run scheduled base backups through nocache, for example behind an opt-in variable like WAL_BACKUP_NOCACHE=1 in https://github.com/railwayapp-templates/postgres-ssl/blob/main/pgbackrest-backup-watcher.sh. Or call posix_fadvise(DONTNEED) on the data files once a backup completes. nocache only drops pages the backup itself brought in, so the cache Postgres actually uses stays put.

  2. A soft limit: expose cgroup v2 memory.high alongside the hard limit. The kernel reclaims cache above it without ever OOM-killing, which removes the bet.

  3. Billing: don't count reclaimable inactive_file toward billed memory (one can always dream 😅)

In August, staff mentioned raising with the Postgres image team whether the image should release this cache at the end of each backup. I couldn't find any follow-up, so I'm asking whether that's still planned?

Thanks!

Awaiting User Response

5 Replies

Railway
BOT

8 hours ago

Thanks for the detailed write-up and the tested proposals. We can't give you a timeline or commit to any of the three changes (a nocache or fadvise step in the image's backup path, a soft memory limit, or excluding reclaimable cache from billing), and we can't share the status of the earlier conversation with the image team.

On the billing side, memory charges are based on what we measure for the workload from the host, and that measurement is what you see in the Metrics tab, so cache the backup leaves behind counts until it is released. Your diagnosis of the cause matches what you'd expect from that measurement, and your proposal to scope a cache release to the pages the backup itself read is a clear, concrete suggestion.

Of the options available today, the service memory limit is the only one that keeps managed PITR, with the trade-off you described.


Status changed to Awaiting User Response Railway • about 8 hours ago


Railway

Thanks for the detailed write-up and the tested proposals. We can't give you a timeline or commit to any of the three changes (a nocache or fadvise step in the image's backup path, a soft memory limit, or excluding reclaimable cache from billing), and we can't share the status of the earlier conversation with the image team. On the billing side, memory charges are based on what we measure for the workload from the host, and that measurement is what you see in the Metrics tab, so cache the backup leaves behind counts until it is released. Your diagnosis of the cause matches what you'd expect from that measurement, and your proposal to scope a cache release to the pages the backup itself read is a clear, concrete suggestion. Of the options available today, the service memory limit is the only one that keeps managed PITR, with the trade-off you described.

58f-ops
OP

7 hours ago

Thanks bot, but hopefully someone from the team can reach out to confirm whether something is planned regarding this issue?


Status changed to Awaiting Railway Response Railway • about 7 hours ago


Railway
BOT

4 hours ago

We can't confirm that any of these changes are planned or give a timeline, so the honest answer is that nothing is committed today: not a cache release step in the image's backup path, not a soft memory limit, and not a billing change for reclaimable cache.

For now, the service memory limit is still the only option that keeps managed PITR in place, with the trade-off you described.


Status changed to Awaiting User Response Railway • about 4 hours ago


Railway

We can't confirm that any of these changes are planned or give a timeline, so the honest answer is that nothing is committed today: not a cache release step in the image's backup path, not a soft memory limit, and not a billing change for reclaimable cache. For now, the service memory limit is still the only option that keeps managed PITR in place, with the trade-off you described.

58f-ops
OP

4 hours ago

Ok so perhaps there is another solution I didn't think of?


Status changed to Awaiting Railway Response Railway • about 4 hours ago


Railway
BOT

4 hours ago

We don't see another option beyond the ones you've already looked at. Managed PITR needs the official postgres-ssl image on a major version tag with no custom start command, so a custom image or a cache-trim start command can't be combined with it. The base backups also run on their own fixed cycle, a weekly full plus a daily differential.

That leaves the service memory limit as the only lever that keeps managed PITR in place, with the trade-off you described.


Status changed to Awaiting User Response Railway • about 4 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...