flatrix1
PROOP
19 days ago
Our production Postgres intermittently freezes completely for tens of seconds. During these episodes trivial, zero-cost operations take 30–130 seconds, including Railway's own internal queries. The container is essentially idle and never hits its CPU quota, the underlying host is massively oversubscribed by other tenants.
Evidence the container is NOT at fault
Measured from inside the Database container:
- cpu.max = 1600000 100000 (16 vCPU quota)
- cpu.stat: nr_throttled 0, throttled_usec 0 — we never hit our own CPU limit
- Container CPU usage: ~0.037 vCPU (idle) during an episode
- memory.current ≈ 300 MB of 16 GB; oom_kill 0
Evidence the host is oversubscribed
/proc/loadavg on the 48-core host:
- 54.46 / 71.91 / 81.19 (one sample)
- 404.48 / 212.80 / 124.02 (after a redeploy relocated us to a different host — load far above 48 cores)
Impact in Postgres logs (all UTC)
- 2026-06-18 21:43:41 checkpoint complete: write=3.930s, sync=0.082s, total=61.596s disk was instant; the checkpointer simply wasn't scheduled for ~57s
- 2026-06-18 21:43:36 duration: 54962 ms parse of a plain SELECT (parsing is pure CPU, no I/O, no locks)
- After a redeploy (DB restarted 2026-06-18 22:01):
- duration: 132704 ms — SELECT pg_logical_emit_message(false, 'rwy_pitr_heartbeat', '')
- duration: 96997 ms — SELECT schema_name FROM information_schema.schemata
- duration: 52629 ms — SELECT pg_is_in_recovery()
A 132-second no-op write (pg_logical_emit_message) with the container idle and nr_throttled=0 can only be CPU starvation at the host
scheduler level.
Requests
- Please move this Database service to a non-oversubscribed host / confirm what's saturating the underlying host.
- Redeploying is not a fix, it relocated us to a worse host (load 404). Is there a path to dedicated/reserved CPU for this service?
- If host oversubscription on this tier is expected, please advise the right plan or product to guarantee CPU for a primary database.
0 Replies