Clarification on Resource Limits and Metrics in Sum Mode for Replicated Services
sanket-sleek
PROOP

a month ago

I’m looking for some clarification regarding how resource limits are represented in the metrics dashboard, particularly when using Sum mode.

I have a service configured with 10 replicas, where each replica is configured with:

  • 24 vCPU
  • 24 GB RAM

Based on the documentation, my understanding is that these resources are allocated per replica. Therefore, with 10 replicas, the aggregate resource limit should theoretically be:

  • 240 vCPU
  • 240 GB RAM

However, when I view the metrics graph in Sum mode, the displayed limit remains 24 vCPU / 24 GB RAM, rather than reflecting the aggregate limit across all 10 replicas.

My understanding of Sum mode is that it should aggregate the resource consumption across all replicas. If that is the case, I would also expect the displayed resource limit to be the sum of the limits of all replicas.

This becomes particularly confusing when looking at memory usage. I have observed an instance where the total memory consumption shown was approximately 30 GB, while the graph indicated that this had exceeded the 24 GB limit. When I checked the memory usage per replica, none of them had crossed 10GB individually.

This raises an important question:

Is the 24 GB memory limit shown in the Sum-mode graph a per-replica limit, or is it actually the total memory limit for the entire service?

The reason I’m particularly concerned about this is that I’m running Playwright tests against this service, and the test execution is frequently being terminated unexpectedly due to what appears to be memory exhaustion.

If each of the 10 replicas genuinely has a 24 GB memory limit, I would expect the service to have an aggregate capacity of approximately 240 GB, in which case seeing the workload fail due to memory exhaustion would be unexpected based on the metrics.

So there seems to be a discrepancy somewhere between the configured replica resources, the limits displayed in the metrics dashboard, and the actual resource availability/behaviour.

Could you please clarify:

  1. Whether the configured 24 GB / 24 vCPU limits are per replica or apply to the service as a whole.
  2. How the Sum mode calculates and displays resource limits.
  3. Whether the 24 GB limit shown in Sum mode is expected behaviour or a metrics/dashboard issue.
  4. How I can determine the actual aggregate memory limit and memory consumption across all replicas.
  5. Whether the apparent memory exhaustion causing my Playwright test executions to be terminated could be related to this configuration or metrics discrepancy.

At the moment, the metrics are quite confusing and, based on the behaviour I’m seeing, appear to be inconsistent with my understanding of the configured resources.

I’d appreciate your help in understanding what is actually happening here and, if possible, how I can accurately monitor the resource limits and consumption of a replicated service.

Thank you.

Awaiting Conductor Response$20 Bounty

6 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 1 month ago


vagacerta2026
FREE

a month ago

Railway resource limits (e.g., 24 GB RAM / 24 vCPU) apply strictly per replica, not as a shared global pool. In Sum mode, the dashboard aggregates total usage while displaying the individual replica's limit line, creating a false visual impression of a threshold breach. Playwright crashes are typically caused by load spikes hitting a single replica's isolated limit, triggering an Out-Of-Memory (OOM) termination. To troubleshoot, use Average mode for accurate monitoring, restrict Playwright parallel workers, or increase replica count.


  1. The 24 applies to a replica. (https://railway.com/pricing)

24 GB/vCPU allocated to the service means it can use up to 24 GB/vCPU. However, the y-axis of the graph will only display the limit of a single replica (24 for your case). For the aggregate memory limit across all replicas, it's still 24 GB limits across each replicas, not a big 240 pool. And no, these configurations wouldn't cause process terminations. It's also possible that the terminations are caused by PID exhaustions.


vagacerta2026

Railway resource limits (e.g., 24 GB RAM / 24 vCPU) apply strictly per replica, not as a shared global pool. In Sum mode, the dashboard aggregates total usage while displaying the individual replica's limit line, creating a false visual impression of a threshold breach. Playwright crashes are typically caused by load spikes hitting a single replica's isolated limit, triggering an Out-Of-Memory (OOM) termination. To troubleshoot, use Average mode for accurate monitoring, restrict Playwright parallel workers, or increase replica count.

sanket-sleek
PROOP

a month ago

Thats correct, its an Out-Of-Memory (OOM) termination, but the problem is observability on railway is poor. It does not reflects correct data

Railway says the max capacity is per resource, but their metrics graph do not really share reliable data

Railway as a platform seems less reliable to me, with all the recent outages and now this concern

Thanks buddy for the response, I appreciate!


0x5b62656e5d

1. The 24 applies to a replica. (https://railway.com/pricing) 24 GB/vCPU allocated to the service means it can use up to 24 GB/vCPU. However, the y-axis of the graph will only display the limit of a single replica (24 for your case). For the aggregate memory limit across all replicas, it's still 24 GB limits across each replicas, not a big 240 pool. And no, these configurations wouldn't cause process terminations. It's also possible that the terminations are caused by PID exhaustions.

sanket-sleek
PROOP

a month ago

The y-axis on the graph should be fixed OR REMOVED, its not really helpful in the current state, don't you think so?

In my case I'm executing playwright tests, and they are aborting abruptly (on each of the replica) during a session. The observed is "termination due to Out-Of-Memory (OOM)", when I checked memory usage of each replica, none of them crossed the limit. However, the combined usage did cross the threshold.

That's where all the confusion is stemming from, like if 24vCPU/GB is the max available resource for EACH replica, then how come it terminates, even when none of the replica never crossed the limit

Can you help me with "It's also possible that the terminations are caused by PID exhaustions.", --> like how do I find what PIDs are consuming how much of resource, I don't know where can I see this information, I tried SSHing into the replica, but its hell slow

Thanks buddy for the response, I appreciate!


anakmerch-dot
HOBBY

a month ago

The important part here is that the combined memory usage crossing 24 GB should not itself trigger an OOM if the replicas are isolated. I'd stop using the Sum graph to diagnose the termination and inspect the replica that actually died.

Since you're running Playwright, I'd check PID exhaustion as well as the cgroup OOM counter. Playwright/Chromium can create a surprising number of child processes, especially with parallel workers and repeated browser launches.

Inside one replica, try:

echo "pids current/limit:"
cat /sys/fs/cgroup/pids.current
cat /sys/fs/cgroup/pids.max

echo "memory events:"
cat /sys/fs/cgroup/memory.events

echo "top process counts:"
ps -e -o comm= | sort | uniq -c | sort -nr | head -20

memory.events is particularly useful because it should show whether the cgroup actually recorded an OOM:

oom
oom_kill

If oom_kill increments, then it really was memory pressure on that specific replica/cgroup, regardless of what the aggregated graph showed.

If memory looks comfortably below 24 GB but pids.current is approaching pids.max, then PID exhaustion becomes a much stronger explanation. Chromium is a good candidate here because each browser can spawn multiple renderer/GPU/utility processes.

I'd also log these values periodically from the test runner rather than checking manually over SSH. That should make it possible to correlate the exact moment Playwright dies with either:

  • memory.current approaching memory.max
  • an increment in memory.events
  • pids.current approaching pids.max

For Playwright specifically, I'd also check whether every test/session is properly closing its browser/context/page. A slow process leak can look like an OOM problem even when the RAM graph doesn't show an obvious spike.

Something like:

await context.close();
await browser.close();

after each isolated session is worth verifying.

The key is to inspect the cgroup counters on an individual replica. The Railway aggregate graph can be confusing here, but memory.events and pids.current should tell you what the kernel/container actually enforced.


anakmerch-dot

The important part here is that the combined memory usage crossing 24 GB should not itself trigger an OOM if the replicas are isolated. I'd stop using the Sum graph to diagnose the termination and inspect the replica that actually died. Since you're running Playwright, I'd check PID exhaustion as well as the cgroup OOM counter. Playwright/Chromium can create a surprising number of child processes, especially with parallel workers and repeated browser launches. Inside one replica, try: ```sh echo "pids current/limit:" cat /sys/fs/cgroup/pids.current cat /sys/fs/cgroup/pids.max echo "memory events:" cat /sys/fs/cgroup/memory.events echo "top process counts:" ps -e -o comm= | sort | uniq -c | sort -nr | head -20 ``` `memory.events` is particularly useful because it should show whether the cgroup actually recorded an OOM: ```text oom oom_kill ``` If `oom_kill` increments, then it really was memory pressure on that specific replica/cgroup, regardless of what the aggregated graph showed. If memory looks comfortably below 24 GB but `pids.current` is approaching `pids.max`, then PID exhaustion becomes a much stronger explanation. Chromium is a good candidate here because each browser can spawn multiple renderer/GPU/utility processes. I'd also log these values periodically from the test runner rather than checking manually over SSH. That should make it possible to correlate the exact moment Playwright dies with either: * `memory.current` approaching `memory.max` * an increment in `memory.events` * `pids.current` approaching `pids.max` For Playwright specifically, I'd also check whether every test/session is properly closing its browser/context/page. A slow process leak can look like an OOM problem even when the RAM graph doesn't show an obvious spike. Something like: ```js await context.close(); await browser.close(); ``` after each isolated session is worth verifying. The key is to inspect the cgroup counters on an individual replica. The Railway aggregate graph can be confusing here, but `memory.events` and `pids.current` should tell you what the kernel/container actually enforced.

sanket-sleek
PROOP

a month ago

Hey, thanks much for the detailed reply, this is so helpful, I'll look close into memory usage

Thank you so much, I appreciate!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...