a month ago
I have a small database (~2GB, ~5M rows) that receives maybe one or two queries a second max. For some reason, it is gobbling up memory like crazy and I'm unsure what I can do to lower that usage, aside from limiting the node's RAM (which might OOM it).
I know that there is shared_memory and things like that to configure, but I can't really figure out how to do this in the Railway ecosystem.
Any help would be appreciated
Cheers!
Attachments
Pinned Solution
a month ago
Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use
10 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
I forgot to mention, I'm using the standard https://github.com/railwayapp-templates/postgres-ssl/pkgs/container/postgres-ssl image provided by Railway without any modifications.
The commands you asked for returned the following btw
PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
hash_mem_multiplier 2 NULL
maintenance_work_mem 65536 kB
max_connections 500 NULL
shared_buffers 16384 8kB
work_mem 4096 kB
14 1 5
a month ago
That output rules out the obvious oversized settings: shared_buffers is only 128 MB, work_mem is 4 MB, and maintenance_work_mem is 64 MB.
The one setting I would question is max_connections = 500. PostgreSQL creates a backend process for each connection, so 500 is a lot of possible memory overhead for a database receiving only one or two queries per second. Your current session count looks low, though, so it probably does not explain the entire 1.3 GB baseline by itself.
I would check the actual process/cache breakdown after a restart:
free -h
ps -eo pid,rss,cmd | grep '[p]ostgres'
df -h /dev/shmAlso check whether the Railway logs contain OOM/restart messages. If memory falls after restart and then gradually rises while remaining stable, that is most likely PostgreSQL/Linux caching rather than a leak.
If the application does not need hundreds of simultaneous database sessions, consider lowering max_connections to something like 50–100, but only after confirming the application pool sizes. This setting requires a restart:
ALTER SYSTEM SET max_connections = 100;Railway documents ALTER SYSTEM for PostgreSQL configuration changes, and its PgBouncer option can help when many client connections are involved. Railway PostgreSQL configuration · Railway PgBouncer guide
a month ago
I think the missing piece here is distinguishing PostgreSQL's actual memory usage from Linux page cache / container memory accounting.
Based on the settings you posted, PostgreSQL itself is not configured to aggressively pre-allocate memory:
shared_buffers = 128 MBwork_mem = 4 MBmaintenance_work_mem = 64 MB
So those settings don't explain a large idle baseline.
A ~1.3 GB memory reading for a PostgreSQL container with a ~2 GB database also isn't necessarily abnormal. PostgreSQL relies heavily on the OS page cache, and depending on how Railway's container metrics account for cached pages, filesystem cache may appear as container RAM usage even though it is reclaimable under memory pressure.
Before lowering RAM limits, I'd check this from inside the container:
free -h
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.statThe important values in memory.stat are especially:
anon
file
slab
inactive_file
active_fileIf most of the usage is file / inactive_file, you're primarily looking at filesystem page cache rather than PostgreSQL backend memory. That memory is generally reclaimable.
I'd also check PostgreSQL's actual process RSS:
ps -eo pid,rss,cmd --sort=-rss | grep '[p]ostgres'And current connection count:
SELECT count(*) FROM pg_stat_activity;
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;One thing I would still change is max_connections = 500. Even if you're currently only handling 1–2 queries/sec, this leaves PostgreSQL configured for a very large number of potential backend processes. It probably isn't causing the current 1.3 GB baseline if connection count is low, but it's unnecessarily high and can become expensive under connection spikes.
The more useful diagnostic would be a restart experiment:
- Record Railway RAM usage immediately after restart.
- Record
memory.stat. - Leave the DB mostly idle.
- Run a sequential scan / workload that reads a significant portion of the DB.
- Observe whether memory rises mainly in
filecache. - Apply memory pressure (or wait for the host/container to need memory) and see whether cache is reclaimed.
If RAM grows with database reads but anon stays relatively flat while file grows, that's expected page-cache behavior rather than a leak.
I would not cap the container RAM until checking this distinction. A hard limit could turn harmless reclaimable cache into OOM kills.
My current hypothesis: the Railway graph is showing cgroup memory that includes filesystem cache, making normal PostgreSQL caching look like excessive RAM usage.
piyush142
That output rules out the obvious oversized settings: `shared_buffers` is only 128 MB, `work_mem` is 4 MB, and `maintenance_work_mem` is 64 MB. The one setting I would question is `max_connections = 500`. PostgreSQL creates a backend process for each connection, so 500 is a lot of possible memory overhead for a database receiving only one or two queries per second. Your current session count looks low, though, so it probably does not explain the entire 1.3 GB baseline by itself. I would check the actual process/cache breakdown after a restart: ```sh free -h ps -eo pid,rss,cmd | grep '[p]ostgres' df -h /dev/shm ``` Also check whether the Railway logs contain OOM/restart messages. If memory falls after restart and then gradually rises while remaining stable, that is most likely PostgreSQL/Linux caching rather than a leak. If the application does not need hundreds of simultaneous database sessions, consider lowering `max_connections` to something like 50–100, but only after confirming the application pool sizes. This setting requires a restart: ```sql ALTER SYSTEM SET max_connections = 100; ``` Railway documents `ALTER SYSTEM` for PostgreSQL configuration changes, and its PgBouncer option can help when many client connections are involved. [Railway PostgreSQL configuration](https://docs.railway.com/databases/postgresql) · [Railway PgBouncer guide](https://docs.railway.com/guides/connection-pooling-pgbouncer)
a month ago
Nothing here jumps out at me. I have a feeling it is caching, because I am getting a 99.78% cache hit regardless of usage. I have absolutely no need for such high level of caching.
root@02d8d33f1a96:/# ps -eo pid,rss,cmd | grep '[p]ostgres'
1 744 /run/podman-init -- /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
2 1644 /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
3 4544 /bin/bash /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
41 49760 postgres -p 5432 -c listen_addresses=*
64 138600 postgres: io worker 0
65 136916 postgres: io worker 1
66 134780 postgres: io worker 2
67 6552 postgres: checkpointer
68 73964 postgres: background writer
70 10984 postgres: walwriter
71 10512 postgres: autovacuum launcher
72 9028 postgres: logical replication launcher
138 161340 postgres: postgres railway 10.146.196.113(36152) idle
145 137484 postgres: postgres railway 10.146.196.113(35988) idle
146 156852 postgres: postgres railway 10.146.196.113(36000) idle
root@02d8d33f1a96:/# free -h
total used free shared buff/cache available
Mem: 322Gi 200Gi 44Gi 10Gi 91Gi 121Gi
Swap: 223Gi 20Gi 203Gi
root@02d8d33f1a96:/# ps -eo pid,rss,cmd | grep '[p]ostgres'
1 744 /run/podman-init -- /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
2 1644 /usr/bin/tini -g -- /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
3 4544 /bin/bash /usr/local/bin/wrapper.sh postgres -p 5432 -c listen_addresses=*
41 49760 postgres -p 5432 -c listen_addresses=*
64 138632 postgres: io worker 0
65 136956 postgres: io worker 1
66 134844 postgres: io worker 2
67 6552 postgres: checkpointer
68 89932 postgres: background writer
70 10984 postgres: walwriter
71 10512 postgres: autovacuum launcher
72 9028 postgres: logical replication launcher
138 161684 postgres: postgres railway 10.146.196.113(36152) idle
145 146300 postgres: postgres railway 10.146.196.113(35988) idle
146 157336 postgres: postgres railway 10.146.196.113(36000) idle
root@02d8d33f1a96:/# df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
shm 62M 1.3M 60M 3% /dev/shm
anakmerch-dot
I think the missing piece here is distinguishing PostgreSQL's actual memory usage from Linux page cache / container memory accounting. Based on the settings you posted, PostgreSQL itself is not configured to aggressively pre-allocate memory: * `shared_buffers = 128 MB` * `work_mem = 4 MB` * `maintenance_work_mem = 64 MB` So those settings don't explain a large idle baseline. A ~1.3 GB memory reading for a PostgreSQL container with a ~2 GB database also isn't necessarily abnormal. PostgreSQL relies heavily on the OS page cache, and depending on how Railway's container metrics account for cached pages, filesystem cache may appear as container RAM usage even though it is reclaimable under memory pressure. Before lowering RAM limits, I'd check this from inside the container: ```sh free -h cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.stat ``` The important values in `memory.stat` are especially: ```text anon file slab inactive_file active_file ``` If most of the usage is `file` / `inactive_file`, you're primarily looking at filesystem page cache rather than PostgreSQL backend memory. That memory is generally reclaimable. I'd also check PostgreSQL's actual process RSS: ```sh ps -eo pid,rss,cmd --sort=-rss | grep '[p]ostgres' ``` And current connection count: ```sql SELECT count(*) FROM pg_stat_activity; SELECT state, count(*) FROM pg_stat_activity GROUP BY state; ``` One thing I would still change is `max_connections = 500`. Even if you're currently only handling 1–2 queries/sec, this leaves PostgreSQL configured for a very large number of potential backend processes. It probably isn't causing the current 1.3 GB baseline if connection count is low, but it's unnecessarily high and can become expensive under connection spikes. The more useful diagnostic would be a restart experiment: 1. Record Railway RAM usage immediately after restart. 2. Record `memory.stat`. 3. Leave the DB mostly idle. 4. Run a sequential scan / workload that reads a significant portion of the DB. 5. Observe whether memory rises mainly in `file` cache. 6. Apply memory pressure (or wait for the host/container to need memory) and see whether cache is reclaimed. If RAM grows with database reads but `anon` stays relatively flat while `file` grows, that's expected page-cache behavior rather than a leak. I would not cap the container RAM until checking this distinction. A hard limit could turn harmless reclaimable cache into OOM kills. My current hypothesis: the Railway graph is showing cgroup memory that includes filesystem cache, making normal PostgreSQL caching look like excessive RAM usage.
a month ago
My DB is internal to the network and I know it will never go past 3 connections, I will be lowering this down to 10 either way.
With my services shut down the db only uses ~50Mb. The second drop to 33Mb is after a restart to set max connections to 10.
After one query that requires a large index scan it went up to 333Mb, so I can confidently say all this mem usage comes from caching only.
Now the question becomes, how do I limit this? This memory usage is chewing through my credits and might limit my ability to stick with Railway long-term :(
Railway only gives me the option between 1Gb and 2Gb on the node, so I will be forced to limit to 1Gb and pray it doesn't OOM.
Attachments
a month ago
Sorry, that doesn't help at all. Also, I can see that the cost breakdown on my billing is 99% memory usage. I don't see why that is being called into question...
a month ago
It's alright, what would really help right now would be a vanilla pancacke recipe, it would surely give me the energy to reduce credit usage
a month ago
Nobody has truly "solved" the problem, but I will accept the solution that did help me make sure my theory was right. For the record, I limited the postgres container to 1Gb and the memory usage is stuck at a neat 100% now, but no OOMs after 2 days working. It will still go past the $5 hobby plan, so I will be taking a look at migrating to MySQL (lower memory usage, allegedly) or moving out of Railway (despite loving the ease of setup).
a month ago
Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use
Status changed to Awaiting User Response Railway • about 1 month ago
dev
Heads up, if you do migrate to MySQL, consider MariaDB instead, it has better memory usage than MySQL and is pretty much a drop in replacement. Brody (Railway employee) has a MariaDb template which you could use
a month ago
Actually super helpful, thanks. I will be looking into it.
Side side note, I disabled the plugin pg_stat_statements and memory usage dropped to ~500Mb (actually didn't notice I had this enabled). If anyone hits this problem in the future, this should be the first thing to look at.
Status changed to Awaiting Conductor Response Railway • about 1 month ago
Status changed to Open medim • about 1 month ago
Status changed to Solved medim • about 1 month ago
