Postgres high RAM usage
minoxs
PROOP

a month ago

I have a small database (~2GB, ~5M rows) that receives maybe one or two queries a second max. For some reason, it is gobbling up memory like crazy and I'm unsure what I can do to lower that usage, aside from limiting the node's RAM (which might OOM it).

I know that there is shared_memory and things like that to configure, but I can't really figure out how to do this in the Railway ecosystem.

Any help would be appreciated

Cheers!

image.png

Attachments

Solved$10 Bounty

Pinned Solution

a month ago

Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use

10 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 1 month ago


minoxs
PROOP

a month ago

I forgot to mention, I'm using the standard https://github.com/railwayapp-templates/postgres-ssl/pkgs/container/postgres-ssl image provided by Railway without any modifications.

The commands you asked for returned the following btw


PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit

hash_mem_multiplier	2	NULL

maintenance_work_mem	65536	kB

max_connections	500	NULL

shared_buffers	16384	8kB

work_mem	4096	kB

14	1	5

piyush142
FREE

a month ago

That output rules out the obvious oversized settings: shared_buffers is only 128 MB, work_mem is 4 MB, and maintenance_work_mem is 64 MB.

The one setting I would question is max_connections = 500. PostgreSQL creates a backend process for each connection, so 500 is a lot of possible memory overhead for a database receiving only one or two queries per second. Your current session count looks low, though, so it probably does not explain the entire 1.3 GB baseline by itself.

I would check the actual process/cache breakdown after a restart:

free -h
ps -eo pid,rss,cmd | grep '[p]ostgres'
df -h /dev/shm

Also check whether the Railway logs contain OOM/restart messages. If memory falls after restart and then gradually rises while remaining stable, that is most likely PostgreSQL/Linux caching rather than a leak.

If the application does not need hundreds of simultaneous database sessions, consider lowering max_connections to something like 50–100, but only after confirming the application pool sizes. This setting requires a restart:

ALTER SYSTEM SET max_connections = 100;

Railway documents ALTER SYSTEM for PostgreSQL configuration changes, and its PgBouncer option can help when many client connections are involved. Railway PostgreSQL configuration · Railway PgBouncer guide


anakmerch-dot
HOBBY

a month ago

I think the missing piece here is distinguishing PostgreSQL's actual memory usage from Linux page cache / container memory accounting.

Based on the settings you posted, PostgreSQL itself is not configured to aggressively pre-allocate memory:

  • shared_buffers = 128 MB
  • work_mem = 4 MB
  • maintenance_work_mem = 64 MB

So those settings don't explain a large idle baseline.

A ~1.3 GB memory reading for a PostgreSQL container with a ~2 GB database also isn't necessarily abnormal. PostgreSQL relies heavily on the OS page cache, and depending on how Railway's container metrics account for cached pages, filesystem cache may appear as container RAM usage even though it is reclaimable under memory pressure.

Before lowering RAM limits, I'd check this from inside the container:

free -h
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.stat

The important values in memory.stat are especially:

anon
file
slab
inactive_file
active_file

If most of the usage is file / inactive_file, you're primarily looking at filesystem page cache rather than PostgreSQL backend memory. That memory is generally reclaimable.

I'd also check PostgreSQL's actual process RSS:

ps -eo pid,rss,cmd --sort=-rss | grep '[p]ostgres'

And current connection count:

SELECT count(*) FROM pg_stat_activity;

SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;

One thing I would still change is max_connections = 500. Even if you're currently only handling 1–2 queries/sec, this leaves PostgreSQL configured for a very large number of potential backend processes. It probably isn't causing the current 1.3 GB baseline if connection count is low, but it's unnecessarily high and can become expensive under connection spikes.

The more useful diagnostic would be a restart experiment:

  1. Record Railway RAM usage immediately after restart.
  2. Record memory.stat.
  3. Leave the DB mostly idle.
  4. Run a sequential scan / workload that reads a significant portion of the DB.
  5. Observe whether memory rises mainly in file cache.
  6. Apply memory pressure (or wait for the host/container to need memory) and see whether cache is reclaimed.

If RAM grows with database reads but anon stays relatively flat while file grows, that's expected page-cache behavior rather than a leak.

I would not cap the container RAM until checking this distinction. A hard limit could turn harmless reclaimable cache into OOM kills.

My current hypothesis: the Railway graph is showing cgroup memory that includes filesystem cache, making normal PostgreSQL caching look like excessive RAM usage.


piyush142

That output rules out the obvious oversized settings: `shared_buffers` is only 128 MB, `work_mem` is 4 MB, and `maintenance_work_mem` is 64 MB. The one setting I would question is `max_connections = 500`. PostgreSQL creates a backend process for each connection, so 500 is a lot of possible memory overhead for a database receiving only one or two queries per second. Your current session count looks low, though, so it probably does not explain the entire 1.3 GB baseline by itself. I would check the actual process/cache breakdown after a restart: ```sh free -h ps -eo pid,rss,cmd | grep '[p]ostgres' df -h /dev/shm ``` Also check whether the Railway logs contain OOM/restart messages. If memory falls after restart and then gradually rises while remaining stable, that is most likely PostgreSQL/Linux caching rather than a leak. If the application does not need hundreds of simultaneous database sessions, consider lowering `max_connections` to something like 50–100, but only after confirming the application pool sizes. This setting requires a restart: ```sql ALTER SYSTEM SET max_connections = 100; ``` Railway documents `ALTER SYSTEM` for PostgreSQL configuration changes, and its PgBouncer option can help when many client connections are involved. [Railway PostgreSQL configuration](https://docs.railway.com/databases/postgresql) · [Railway PgBouncer guide](https://docs.railway.com/guides/connection-pooling-pgbouncer)

minoxs
PROOP

a month ago

Nothing here jumps out at me. I have a feeling it is caching, because I am getting a 99.78% cache hit regardless of usage. I have absolutely no need for such high level of caching.


root@02d8d33f1a96:/# ps -eo pid,rss,cmd | grep '[p]ostgres'

      1   744 /run/podman-init -- /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

      2  1644 /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

      3  4544 /bin/bash /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

     41 49760 postgres -p 5432 -c listen_addresses=*

     64 138600 postgres: io worker 0

     65 136916 postgres: io worker 1

     66 134780 postgres: io worker 2

     67  6552 postgres: checkpointer 

     68 73964 postgres: background writer 

     70 10984 postgres: walwriter 

     71 10512 postgres: autovacuum launcher 

     72  9028 postgres: logical replication launcher 

    138 161340 postgres: postgres railway 10.146.196.113(36152) idle

    145 137484 postgres: postgres railway 10.146.196.113(35988) idle

    146 156852 postgres: postgres railway 10.146.196.113(36000) idle

root@02d8d33f1a96:/# free -h

               total        used        free      shared  buff/cache   available

Mem:           322Gi       200Gi        44Gi        10Gi        91Gi       121Gi

Swap:          223Gi        20Gi       203Gi

root@02d8d33f1a96:/# ps -eo pid,rss,cmd | grep '[p]ostgres'

      1   744 /run/podman-init -- /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

      2  1644 /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

      3  4544 /bin/bash /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*

     41 49760 postgres -p 5432 -c listen_addresses=*

     64 138632 postgres: io worker 0

     65 136956 postgres: io worker 1

     66 134844 postgres: io worker 2

     67  6552 postgres: checkpointer 

     68 89932 postgres: background writer 

     70 10984 postgres: walwriter 

     71 10512 postgres: autovacuum launcher 

     72  9028 postgres: logical replication launcher 

    138 161684 postgres: postgres railway 10.146.196.113(36152) idle

    145 146300 postgres: postgres railway 10.146.196.113(35988) idle

    146 157336 postgres: postgres railway 10.146.196.113(36000) idle

root@02d8d33f1a96:/# df -h /dev/shm

Filesystem      Size  Used Avail Use% Mounted on

shm              62M  1.3M   60M   3% /dev/shm

anakmerch-dot

I think the missing piece here is distinguishing PostgreSQL's actual memory usage from Linux page cache / container memory accounting. Based on the settings you posted, PostgreSQL itself is not configured to aggressively pre-allocate memory: * `shared_buffers = 128 MB` * `work_mem = 4 MB` * `maintenance_work_mem = 64 MB` So those settings don't explain a large idle baseline. A ~1.3 GB memory reading for a PostgreSQL container with a ~2 GB database also isn't necessarily abnormal. PostgreSQL relies heavily on the OS page cache, and depending on how Railway's container metrics account for cached pages, filesystem cache may appear as container RAM usage even though it is reclaimable under memory pressure. Before lowering RAM limits, I'd check this from inside the container: ```sh free -h cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.stat ``` The important values in `memory.stat` are especially: ```text anon file slab inactive_file active_file ``` If most of the usage is `file` / `inactive_file`, you're primarily looking at filesystem page cache rather than PostgreSQL backend memory. That memory is generally reclaimable. I'd also check PostgreSQL's actual process RSS: ```sh ps -eo pid,rss,cmd --sort=-rss | grep '[p]ostgres' ``` And current connection count: ```sql SELECT count(*) FROM pg_stat_activity; SELECT state, count(*) FROM pg_stat_activity GROUP BY state; ``` One thing I would still change is `max_connections = 500`. Even if you're currently only handling 1–2 queries/sec, this leaves PostgreSQL configured for a very large number of potential backend processes. It probably isn't causing the current 1.3 GB baseline if connection count is low, but it's unnecessarily high and can become expensive under connection spikes. The more useful diagnostic would be a restart experiment: 1. Record Railway RAM usage immediately after restart. 2. Record `memory.stat`. 3. Leave the DB mostly idle. 4. Run a sequential scan / workload that reads a significant portion of the DB. 5. Observe whether memory rises mainly in `file` cache. 6. Apply memory pressure (or wait for the host/container to need memory) and see whether cache is reclaimed. If RAM grows with database reads but `anon` stays relatively flat while `file` grows, that's expected page-cache behavior rather than a leak. I would not cap the container RAM until checking this distinction. A hard limit could turn harmless reclaimable cache into OOM kills. My current hypothesis: the Railway graph is showing cgroup memory that includes filesystem cache, making normal PostgreSQL caching look like excessive RAM usage.

minoxs
PROOP

a month ago

My DB is internal to the network and I know it will never go past 3 connections, I will be lowering this down to 10 either way.

With my services shut down the db only uses ~50Mb. The second drop to 33Mb is after a restart to set max connections to 10.

image.png

After one query that requires a large index scan it went up to 333Mb, so I can confidently say all this mem usage comes from caching only.

Now the question becomes, how do I limit this? This memory usage is chewing through my credits and might limit my ability to stick with Railway long-term :(

Railway only gives me the option between 1Gb and 2Gb on the node, so I will be forced to limit to 1Gb and pray it doesn't OOM.

Attachments


minoxs
PROOP

a month ago

Sorry, that doesn't help at all. Also, I can see that the cost breakdown on my billing is 99% memory usage. I don't see why that is being called into question...


minoxs
PROOP

a month ago

It's alright, what would really help right now would be a vanilla pancacke recipe, it would surely give me the energy to reduce credit usage


minoxs
PROOP

a month ago

Nobody has truly "solved" the problem, but I will accept the solution that did help me make sure my theory was right. For the record, I limited the postgres container to 1Gb and the memory usage is stuck at a neat 100% now, but no OOMs after 2 days working. It will still go past the $5 hobby plan, so I will be taking a look at migrating to MySQL (lower memory usage, allegedly) or moving out of Railway (despite loving the ease of setup).


a month ago

Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use


Status changed to Awaiting User Response Railway • about 1 month ago


dev

Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use

minoxs
PROOP

a month ago

Actually super helpful, thanks. I will be looking into it.

Side side note, I disabled the plugin pg_stat_statements and memory usage dropped to ~500Mb (actually didn't notice I had this enabled). If anyone hits this problem in the future, this should be the first thing to look at.


Status changed to Awaiting Conductor Response Railway • about 1 month ago


Status changed to Open medim • about 1 month ago


Status changed to Solved medim • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...