Skip to content
Volume-backed service stuck on an overloaded host, CPU-bound slowdown
legendanish-roy
OP

a month ago

Hi team,

One of our Pro plan services that uses a volume seems to be on an overloaded host, and we’d like it moved to a different machine. I can share the project, service and volume ids privately.

Measured from inside the container:

The CPU is “Intel Xeon Processor (Icelake)”. Host loadavg is about 53 on 48 CPUs, with about 45 processes running.

cgroup cpu.pressure is some=100% and full=96% (avg60), so our threads are almost always waiting for a CPU.

We use about 1.65 of a 3 vCPU limit, and CFS throttling is negligible (6 of 100 periods). The waiting comes from the host, not from our quota.

The volume device is 86% busy, at 2.7 ms per write and 1.9 ms per read.

Two identical services in the same project (same code, same load, also with volumes) run on AMD EPYC 9655P hosts. Each runs at loadavg ~27 with CPU pressure under 1%, and one of their volumes measures 0.8 ms per write. On the Intel host, the same work uses about 1.7x the CPU and runs 5-10x slower. It has been like this for about two weeks, through several redeploys, presumably because the volume keeps the service on that host.

Could you move this service and its volume to a less loaded host? A short restart is fine.

Thanks!

Solved

1 Reply

Status changed to Awaiting Railway Response Railway • 28 days ago


a month ago

We've moved polymarket-capture-0-16 and its volume to a Gen 2 host with an AMD EPYC 9655P, matching the CPU model of your other two services. The service is back up.

Could you check how performance looks now? Thanks for sharing the detailed measurements.


Status changed to Awaiting User Response Railway • 28 days ago


Railway
BOT

17 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 17 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...