a month ago
I’m looking for some clarification regarding how resource limits are represented in the metrics dashboard, particularly when using Sum mode.
I have a service configured with 10 replicas, where each replica is configured with:
- 24 vCPU
- 24 GB RAM
Based on the documentation, my understanding is that these resources are allocated per replica. Therefore, with 10 replicas, the aggregate resource limit should theoretically be:
- 240 vCPU
- 240 GB RAM
However, when I view the metrics graph in Sum mode, the displayed limit remains 24 vCPU / 24 GB RAM, rather than reflecting the aggregate limit across all 10 replicas.
My understanding of Sum mode is that it should aggregate the resource consumption across all replicas. If that is the case, I would also expect the displayed resource limit to be the sum of the limits of all replicas.
This becomes particularly confusing when looking at memory usage. I have observed an instance where the total memory consumption shown was approximately 30 GB, while the graph indicated that this had exceeded the 24 GB limit. When I checked the memory usage per replica, none of them had crossed 10GB individually.
This raises an important question:
Is the 24 GB memory limit shown in the Sum-mode graph a per-replica limit, or is it actually the total memory limit for the entire service?
The reason I’m particularly concerned about this is that I’m running Playwright tests against this service, and the test execution is frequently being terminated unexpectedly due to what appears to be memory exhaustion.
If each of the 10 replicas genuinely has a 24 GB memory limit, I would expect the service to have an aggregate capacity of approximately 240 GB, in which case seeing the workload fail due to memory exhaustion would be unexpected based on the metrics.
So there seems to be a discrepancy somewhere between the configured replica resources, the limits displayed in the metrics dashboard, and the actual resource availability/behaviour.
Could you please clarify:
- Whether the configured 24 GB / 24 vCPU limits are per replica or apply to the service as a whole.
- How the Sum mode calculates and displays resource limits.
- Whether the 24 GB limit shown in Sum mode is expected behaviour or a metrics/dashboard issue.
- How I can determine the actual aggregate memory limit and memory consumption across all replicas.
- Whether the apparent memory exhaustion causing my Playwright test executions to be terminated could be related to this configuration or metrics discrepancy.
At the moment, the metrics are quite confusing and, based on the behaviour I’m seeing, appear to be inconsistent with my understanding of the configured resources.
I’d appreciate your help in understanding what is actually happening here and, if possible, how I can accurately monitor the resource limits and consumption of a replicated service.
Thank you.
6 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Railway resource limits (e.g., 24 GB RAM / 24 vCPU) apply strictly per replica, not as a shared global pool. In Sum mode, the dashboard aggregates total usage while displaying the individual replica's limit line, creating a false visual impression of a threshold breach. Playwright crashes are typically caused by load spikes hitting a single replica's isolated limit, triggering an Out-Of-Memory (OOM) termination. To troubleshoot, use Average mode for accurate monitoring, restrict Playwright parallel workers, or increase replica count.
a month ago
- The 24 applies to a replica. (https://railway.com/pricing)
24 GB/vCPU allocated to the service means it can use up to 24 GB/vCPU. However, the y-axis of the graph will only display the limit of a single replica (24 for your case). For the aggregate memory limit across all replicas, it's still 24 GB limits across each replicas, not a big 240 pool. And no, these configurations wouldn't cause process terminations. It's also possible that the terminations are caused by PID exhaustions.
vagacerta2026
Railway resource limits (e.g., 24 GB RAM / 24 vCPU) apply strictly per replica, not as a shared global pool. In Sum mode, the dashboard aggregates total usage while displaying the individual replica's limit line, creating a false visual impression of a threshold breach. Playwright crashes are typically caused by load spikes hitting a single replica's isolated limit, triggering an Out-Of-Memory (OOM) termination. To troubleshoot, use Average mode for accurate monitoring, restrict Playwright parallel workers, or increase replica count.
a month ago
Thats correct, its an Out-Of-Memory (OOM) termination, but the problem is observability on railway is poor. It does not reflects correct data
Railway says the max capacity is per resource, but their metrics graph do not really share reliable data
Railway as a platform seems less reliable to me, with all the recent outages and now this concern
Thanks buddy for the response, I appreciate!
0x5b62656e5d
1. The 24 applies to a replica. (https://railway.com/pricing) 24 GB/vCPU allocated to the service means it can use up to 24 GB/vCPU. However, the y-axis of the graph will only display the limit of a single replica (24 for your case). For the aggregate memory limit across all replicas, it's still 24 GB limits across each replicas, not a big 240 pool. And no, these configurations wouldn't cause process terminations. It's also possible that the terminations are caused by PID exhaustions.
a month ago
The y-axis on the graph should be fixed OR REMOVED, its not really helpful in the current state, don't you think so?
In my case I'm executing playwright tests, and they are aborting abruptly (on each of the replica) during a session. The observed is "termination due to Out-Of-Memory (OOM)", when I checked memory usage of each replica, none of them crossed the limit. However, the combined usage did cross the threshold.
That's where all the confusion is stemming from, like if 24vCPU/GB is the max available resource for EACH replica, then how come it terminates, even when none of the replica never crossed the limit
Can you help me with "It's also possible that the terminations are caused by PID exhaustions.", --> like how do I find what PIDs are consuming how much of resource, I don't know where can I see this information, I tried SSHing into the replica, but its hell slow
Thanks buddy for the response, I appreciate!
a month ago
The important part here is that the combined memory usage crossing 24 GB should not itself trigger an OOM if the replicas are isolated. I'd stop using the Sum graph to diagnose the termination and inspect the replica that actually died.
Since you're running Playwright, I'd check PID exhaustion as well as the cgroup OOM counter. Playwright/Chromium can create a surprising number of child processes, especially with parallel workers and repeated browser launches.
Inside one replica, try:
echo "pids current/limit:"
cat /sys/fs/cgroup/pids.current
cat /sys/fs/cgroup/pids.max
echo "memory events:"
cat /sys/fs/cgroup/memory.events
echo "top process counts:"
ps -e -o comm= | sort | uniq -c | sort -nr | head -20memory.events is particularly useful because it should show whether the cgroup actually recorded an OOM:
oom
oom_killIf oom_kill increments, then it really was memory pressure on that specific replica/cgroup, regardless of what the aggregated graph showed.
If memory looks comfortably below 24 GB but pids.current is approaching pids.max, then PID exhaustion becomes a much stronger explanation. Chromium is a good candidate here because each browser can spawn multiple renderer/GPU/utility processes.
I'd also log these values periodically from the test runner rather than checking manually over SSH. That should make it possible to correlate the exact moment Playwright dies with either:
memory.currentapproachingmemory.max- an increment in
memory.events pids.currentapproachingpids.max
For Playwright specifically, I'd also check whether every test/session is properly closing its browser/context/page. A slow process leak can look like an OOM problem even when the RAM graph doesn't show an obvious spike.
Something like:
await context.close();
await browser.close();after each isolated session is worth verifying.
The key is to inspect the cgroup counters on an individual replica. The Railway aggregate graph can be confusing here, but memory.events and pids.current should tell you what the kernel/container actually enforced.
anakmerch-dot
The important part here is that the combined memory usage crossing 24 GB should not itself trigger an OOM if the replicas are isolated. I'd stop using the Sum graph to diagnose the termination and inspect the replica that actually died. Since you're running Playwright, I'd check PID exhaustion as well as the cgroup OOM counter. Playwright/Chromium can create a surprising number of child processes, especially with parallel workers and repeated browser launches. Inside one replica, try: ```sh echo "pids current/limit:" cat /sys/fs/cgroup/pids.current cat /sys/fs/cgroup/pids.max echo "memory events:" cat /sys/fs/cgroup/memory.events echo "top process counts:" ps -e -o comm= | sort | uniq -c | sort -nr | head -20 ``` `memory.events` is particularly useful because it should show whether the cgroup actually recorded an OOM: ```text oom oom_kill ``` If `oom_kill` increments, then it really was memory pressure on that specific replica/cgroup, regardless of what the aggregated graph showed. If memory looks comfortably below 24 GB but `pids.current` is approaching `pids.max`, then PID exhaustion becomes a much stronger explanation. Chromium is a good candidate here because each browser can spawn multiple renderer/GPU/utility processes. I'd also log these values periodically from the test runner rather than checking manually over SSH. That should make it possible to correlate the exact moment Playwright dies with either: * `memory.current` approaching `memory.max` * an increment in `memory.events` * `pids.current` approaching `pids.max` For Playwright specifically, I'd also check whether every test/session is properly closing its browser/context/page. A slow process leak can look like an OOM problem even when the RAM graph doesn't show an obvious spike. Something like: ```js await context.close(); await browser.close(); ``` after each isolated session is worth verifying. The key is to inspect the cgroup counters on an individual replica. The Railway aggregate graph can be confusing here, but `memory.events` and `pids.current` should tell you what the kernel/container actually enforced.
a month ago
Hey, thanks much for the detailed reply, this is so helpful, I'll look close into memory usage
Thank you so much, I appreciate!