Volume-backed service stuck on an overloaded host, CPU-bound slowdown
legendanish-roy
PROOP

2 hours ago

Hi team,

One of our Pro plan services that uses a volume seems to be on an overloaded host, and we’d like it moved to a different machine. I can share the project, service and volume ids privately.

Measured from inside the container:

The CPU is “Intel Xeon Processor (Icelake)”. Host loadavg is about 53 on 48 CPUs, with about 45 processes running.

cgroup cpu.pressure is some=100% and full=96% (avg60), so our threads are almost always waiting for a CPU.

We use about 1.65 of a 3 vCPU limit, and CFS throttling is negligible (6 of 100 periods). The waiting comes from the host, not from our quota.

The volume device is 86% busy, at 2.7 ms per write and 1.9 ms per read.

Two identical services in the same project (same code, same load, also with volumes) run on AMD EPYC 9655P hosts. Each runs at loadavg ~27 with CPU pressure under 1%, and one of their volumes measures 0.8 ms per write. On the Intel host, the same work uses about 1.7x the CPU and runs 5-10x slower. It has been like this for about two weeks, through several redeploys, presumably because the volume keeps the service on that host.

Could you move this service and its volume to a less loaded host? A short restart is fine.

Thanks!

Awaiting Railway Response

0 Replies

Status changed to Awaiting Railway Response Railway • about 2 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...