Production outage — severe disk I/O degradation on Postgres volume (DB up but unusable)
contas-ginga
PROOP

a month ago

Our production Postgres service is experiencing severe, sustained disk I/O latency that makes the database effectively unusable. This is a platform/storage issue, not an application-level one — the database itself is healthy and reports ready to accept connections.

Evidence:

fsync of tiny amounts of data takes 10–18 seconds. Postgres checkpoints writing only ~100–700 kB routinely take 80–100 seconds total (sync phase alone 14–18s). Degrading over several days, became a full outage today.

The volume is at 1% usage (354 MB used of 46 GB) — not a capacity issue.

Small DB (~354 MB), minimal write load (checkpoint distance only a few hundred kB per cycle).

Live write test confirms it: dd if=/dev/zero of=/testfile bs=8k count=1000 oflag=dsync (8 MB) does not complete in over a minute. Healthy storage does this in a couple seconds.

We already redeployed the Postgres service. Compute moved to a new host, but I/O latency persists — isolating the problem to the storage volume itself, not the compute host.

Last boot's crash recovery logged syncing data directory (fsync), elapsed time: 11.59 s — abnormal for a tiny data dir.

Downstream impact: connection-pool exhaustion in our app (timeout exceeded when trying to connect in pg-pool), and could not send data to client: Broken pipe on the server side, because queries block on disk and never release connections.

Volume ID / bind-mount: vol_ccbmenazoop2ufbv (bind-mounts/763a382a-4918-4e1c-87cc-36a169233029/...)

Please check I/O contention / storage health on the host backing this volume, and migrate the volume to healthy storage if needed. This is a production outage. Happy to provide full logs.

$20 Bounty

2 Replies

Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 1 month ago


lawrencegripperwrk
EMPLOYEE

a month ago

Hi,

Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.


lawrencegripperwrk

Hi, Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.

contas-ginga
PROOP

a month ago

Thanks a lot for the quick turnaround, Lawrence — really appreciate you tracking down the network link.

I'm seeing improvement: a dd ... oflag=dsync write test went from ~36 kB/s (during the incident) to ~568 kB/s now. That said, it's still well below what I'd expect from healthy storage, so I don't think it's fully back to normal yet. Could you keep an eye on it / confirm whether more recovery is expected as things settle?

I'll keep monitoring on my side and follow up if it degrades again. Thanks again!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...