14 hours ago
- Certain operations use a few GB of memory, but that memory doesn't get released after the operation is complete, keeping mem usage permenantly high
- I think this is due to the containers mounted filesystem cache, which I don't have permission to modify.
Investigation:
I inspected the database container at 2026-09-14 04:26:43 UTC: total memory was 1.990 GB against a 2 GB limit, with 1.695 GB (85%) accounted for by inactive filesystem cache, versus 87.9 MB of anonymous memory and 149.5 MB of shared memory/tmpfs. Only 0.26 MB of file pages were dirty. Even as root, I cannot write to memory.reclaim, memory.high, or /proc/sys/vm/drop_caches because /sys/fs/cgroup and /proc/sys are mounted read-only.
6 Replies
14 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 14 hours ago
4 hours ago
This is standard Linux cgroup memory accounting behavior. In Linux containers, the host kernel automatically uses free RAM to cache disk I/O (page cache). Memory used by inactive page cache is harmless—it stays allocated until the system needs it, at which point the kernel automatically reclaims it for processes without causing Out-Of-Memory (OOM) kills.Because Railway runs containers without elevated privileges, users cannot manually trigger /proc/sys/vm/drop_caches or write to memory.reclaim.Suggested Response to the UserHi @cprosche,What you are observing is normal Linux kernel memory accounting. The high usage you see in metrics is page cache (file system caching), which PostgreSQL/TimescaleDB relies on heavily for fast disk reads.Why This HappensCached RAM is Available RAM: Linux uses unused RAM to cache disk blocks. When an operation finishes, the kernel keeps those pages in memory in case they are read again.Automatic Reclamation: When PostgreSQL or another process requests more memory, the kernel instantly drops or reclaims inactive file cache pages. Cache memory will not cause an OOM (Out Of Memory) crash.Restricted OS Access: Container environments run without root-level host access, which is why /proc/sys/vm/drop_caches and cgroup reclaim files are read-only.What to Do1.Check Anonymous vs. Cache Memory:Monitor your database memory breakdown rather than total overall memory. As long as anonymous memory (actual application state/query working RAM) stays safely below your 2 GB limit, your database is operating normally.2.Tune PostgreSQL Buffer & Work Memory:If you want to reduce peak memory consumption during heavy queries, adjust your PostgreSQL configuration in postgresql.conf or environment variables:Lower work_mem to restrict per-operation query memory.Ensure shared_buffers is set appropriately for a 2 GB container (typically 256MB–512MB).3.Verify Container Stability:Check your deployment metrics or logs during peak activity. You can confirm the cache is reclaiming properly if queries run smoothly without the container restarting due to OOM errors.
4 hours ago
Those measurements point to retained Linux page cache rather than 1.99 GB of live TimescaleDB allocations. Clean filesystem cache can remain resident after a query finishes: Linux keeps it for subsequent reads and normally reclaims it when memory is needed. It does not have to drop immediately when the operation ends.
Using your numbers, 1.990 GB - 1.695 GB ≈ 0.295 GB remains after subtracting inactive file pages. That subtraction is only a rough working-set indicator, not a precise PostgreSQL allocation total or a Railway billing measurement. Also, shmem is included in the broader file accounting, so don't add every memory.stat field together.
The distinction to check is whether this is just a high graph while the database stays healthy, or whether allocations actually trigger stalls/OOMs. Take the following snapshot in the same database cgroup you inspected, before and after the operation and during normal subsequent traffic:
cd /sys/fs/cgroup || exit 1
date -u
for name in memory.current memory.max memory.events memory.pressure; do
if [ -r "$name" ]; then
printf '\n%s\n' "$name"
cat "$name"
fi
done
if [ -r memory.stat ]; then
printf '\nmemory.stat (selected fields; bytes)\n'
awk '$1 ~ /^(anon|file|shmem|inactive_file|active_file|file_dirty|file_writeback|unevictable)$/ { print }' memory.stat
fiRead memory.events as counter changes between snapshots, not as fresh incidents just because a counter is nonzero. Increasing max means the limit was encountered; it does not by itself mean a process was killed. Increasing oom or oom_kill, sustained memory pressure, and application latency/errors are stronger signs that there is a real capacity or reclaim problem.
If clean inactive file cache gives way as the database needs memory and there are no pressure symptoms, there is no cache leak to fix. Read-only cgroup/sysctl mounts prevent container root from using those host controls; they do not prevent the kernel's automatic reclaim. I would not try to remount them, run a scheduled cache flush, or restart a healthy database just to lower the graph.
If the concern is the bill, Railway needs to confirm how its billed RAM is calculated; their metrics page describes RAM consumption but does not specify whether inactive file cache is excluded. These cgroup numbers alone cannot establish that. If OOMs/stalls do occur while most usage remains clean inactive file cache, the before/after snapshots plus the deployment ID and UTC timestamps give Railway evidence to inspect host-side reclaim. PostgreSQL tuning should then follow the actual query/worker memory demand, rather than treating all file cache as shared_buffers or work_mem.
References:
- Linux cgroup v2 memory controller, including
memory.max,memory.events, andmemory.stat: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory - Railway metrics: https://docs.railway.com/observability/metrics
Chandan
2 hours ago
Checking the background-job timestamps is useful. One correction: timescaledb.compress_chunk_time_interval controls merging compressed chunks. It does not split the original chunks or cap a compression job's memory. The compression documentation says each chunk is compressed and may be merged into an adjacent chunk; decompression does not split those chunks again.
For smaller chunks going forward, set_chunk_time_interval() changes the interval for new chunks only. Existing chunks keep their current intervals. Neither setting releases the 1.695 GB of inactive cache already reported here.
I would correlate the job timestamps with changes in memory.events, memory.pressure, and query latency before changing the chunk layout. Matching times are a useful lead, not proof of the cause. Without stalls or OOM events, the retained clean cache alone does not justify a configuration change. If there are symptoms, the installed TimescaleDB version and the specific job configuration will determine which tuning controls apply.
Compression setting:
https://github.com/timescale/docs/blob/latest/api/compression/alter_table_compression.md
Chunk interval and existing-chunk limitation:
https://docs.tigerdata.com/api/latest/hypertable/set_chunk_time_interval/
Chandan
Those measurements point to retained Linux page cache rather than 1.99 GB of live TimescaleDB allocations. Clean filesystem cache can remain resident after a query finishes: Linux keeps it for subsequent reads and normally reclaims it when memory is needed. It does not have to drop immediately when the operation ends. Using your numbers, `1.990 GB - 1.695 GB ≈ 0.295 GB` remains after subtracting inactive file pages. That subtraction is only a rough working-set indicator, not a precise PostgreSQL allocation total or a Railway billing measurement. Also, `shmem` is included in the broader `file` accounting, so don't add every `memory.stat` field together. The distinction to check is whether this is just a high graph while the database stays healthy, or whether allocations actually trigger stalls/OOMs. Take the following snapshot in the **same database cgroup you inspected**, before and after the operation and during normal subsequent traffic: ```sh cd /sys/fs/cgroup || exit 1 date -u for name in memory.current memory.max memory.events memory.pressure; do if [ -r "$name" ]; then printf '\n%s\n' "$name" cat "$name" fi done if [ -r memory.stat ]; then printf '\nmemory.stat (selected fields; bytes)\n' awk '$1 ~ /^(anon|file|shmem|inactive_file|active_file|file_dirty|file_writeback|unevictable)$/ { print }' memory.stat fi ``` Read `memory.events` as **counter changes between snapshots**, not as fresh incidents just because a counter is nonzero. Increasing `max` means the limit was encountered; it does not by itself mean a process was killed. Increasing `oom` or `oom_kill`, sustained memory pressure, and application latency/errors are stronger signs that there is a real capacity or reclaim problem. If clean inactive file cache gives way as the database needs memory and there are no pressure symptoms, there is no cache leak to fix. Read-only cgroup/sysctl mounts prevent container root from using those host controls; they do not prevent the kernel's automatic reclaim. I would not try to remount them, run a scheduled cache flush, or restart a healthy database just to lower the graph. If the concern is the **bill**, Railway needs to confirm how its billed RAM is calculated; their metrics page describes RAM consumption but does not specify whether inactive file cache is excluded. These cgroup numbers alone cannot establish that. If OOMs/stalls do occur while most usage remains clean inactive file cache, the before/after snapshots plus the deployment ID and UTC timestamps give Railway evidence to inspect host-side reclaim. PostgreSQL tuning should then follow the actual query/worker memory demand, rather than treating all file cache as `shared_buffers` or `work_mem`. References: - Linux cgroup v2 memory controller, including `memory.max`, `memory.events`, and `memory.stat`: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory - Railway metrics: https://docs.railway.com/observability/metrics Chandan
an hour ago
The concern is the bill, which according to my math, I’m being billed for inactive file cache as memory, which is EXPENSIVE. Sorry that wasn’t clearer in my original post.
cprosche
The concern is the bill, which according to my math, I’m being billed for inactive file cache as memory, which is EXPENSIVE. Sorry that wasn’t clearer in my original post.
an hour ago
Thanks for clarifying. Reclaimable memory can still be a cost problem if it remains resident and is included in the usage meter. Automatic reclaim under pressure does not mean it will disappear during idle time.
At Railway's published RAM rate of $10/GB/month, a sustained, metered 1.695 GB would contribute about $16.95/month in memory usage before included usage and taxes. That is conditional on the cache actually being counted; the public metrics documentation does not specify the exact cgroup accounting formula.
There is a Linux mechanism worth testing for the permissions problem: per-file posix_fadvise(..., POSIX_FADV_DONTNEED). It asks the kernel to discard cached pages for a readable file, without writing to /proc/sys or /sys/fs/cgroup. I checked this on a disposable 16 MiB file under Linux: resident pages went from 4096 to 0. That is a local syscall test, not a Railway volume or billing test.
If Python is already available, first try this only on a disposable, already-read test file on a staging volume:
import os
fd = os.open("/path/to/disposable-test-file", os.O_RDONLY)
try:
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
finally:
os.close(fd)It is advisory: unwritten dirty pages are not freed, some filesystem implementations may ignore it, and subsequent reads can populate the cache again. I would not turn this into a recursive live-PGDATA flush or scheduled production job; that needs a separate performance assessment.
The documented Railway control is service Settings → Deploy → Replica Limits → memory. A smaller total-memory ceiling can constrain how much file cache accumulates, but it also caps PostgreSQL and can cause OOMs. Choose it from representative peak-load testing with headroom, not the single 0.295 GB subtraction. Changing effective_cache_size only changes planner estimates; it does not cap the kernel cache.
For either experiment, compare the service's memory usage charge over matching time windows as well as memory.current/memory.stat. A lower Linux cache reading alone is not proof of a lower invoice. Railway still needs to confirm whether its billed-memory calculation includes inactive_file on this runtime.
References:
https://man7.org/linux/man-pages/man2/posix_fadvise.2.html
https://docs.railway.com/guides/right-size-cpu-memory
https://docs.railway.com/pricing/cost-control#replica-limits
Chandan
Thanks for clarifying. Reclaimable memory can still be a cost problem if it remains resident and is included in the usage meter. Automatic reclaim under pressure does not mean it will disappear during idle time. At Railway's published RAM rate of $10/GB/month, a sustained, metered 1.695 GB would contribute about $16.95/month in memory usage before included usage and taxes. That is conditional on the cache actually being counted; the public metrics documentation does not specify the exact cgroup accounting formula. There is a Linux mechanism worth testing for the permissions problem: per-file `posix_fadvise(..., POSIX_FADV_DONTNEED)`. It asks the kernel to discard cached pages for a readable file, without writing to `/proc/sys` or `/sys/fs/cgroup`. I checked this on a disposable 16 MiB file under Linux: resident pages went from 4096 to 0. That is a local syscall test, not a Railway volume or billing test. If Python is already available, first try this only on a disposable, already-read test file on a staging volume: ```python import os fd = os.open("/path/to/disposable-test-file", os.O_RDONLY) try: os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) finally: os.close(fd) ``` It is advisory: unwritten dirty pages are not freed, some filesystem implementations may ignore it, and subsequent reads can populate the cache again. I would not turn this into a recursive live-PGDATA flush or scheduled production job; that needs a separate performance assessment. The documented Railway control is **service Settings → Deploy → Replica Limits → memory**. A smaller total-memory ceiling can constrain how much file cache accumulates, but it also caps PostgreSQL and can cause OOMs. Choose it from representative peak-load testing with headroom, not the single 0.295 GB subtraction. Changing `effective_cache_size` only changes planner estimates; it does not cap the kernel cache. For either experiment, compare the service's memory usage charge over matching time windows as well as `memory.current`/`memory.stat`. A lower Linux cache reading alone is not proof of a lower invoice. Railway still needs to confirm whether its billed-memory calculation includes `inactive_file` on this runtime. References: https://man7.org/linux/man-pages/man2/posix_fadvise.2.html https://docs.railway.com/guides/right-size-cpu-memory https://docs.railway.com/pricing/cost-control#replica-limits Chandan
43 minutes ago
I tried restricting the memory aggressively and was getting OOM Kills. With limit at 32 GB, my memory usage was sitting at a sustained ~5 GB.