a month ago
Our production Postgres service is experiencing severe, sustained disk I/O latency that makes the database effectively unusable. This is a platform/storage issue, not an application-level one — the database itself is healthy and reports ready to accept connections.
Evidence:
fsync of tiny amounts of data takes 10–18 seconds. Postgres checkpoints writing only ~100–700 kB routinely take 80–100 seconds total (sync phase alone 14–18s). Degrading over several days, became a full outage today.
The volume is at 1% usage (354 MB used of 46 GB) — not a capacity issue.
Small DB (~354 MB), minimal write load (checkpoint distance only a few hundred kB per cycle).
Live write test confirms it: dd if=/dev/zero of=/testfile bs=8k count=1000 oflag=dsync (8 MB) does not complete in over a minute. Healthy storage does this in a couple seconds.
We already redeployed the Postgres service. Compute moved to a new host, but I/O latency persists — isolating the problem to the storage volume itself, not the compute host.
Last boot's crash recovery logged syncing data directory (fsync), elapsed time: 11.59 s — abnormal for a tiny data dir.
Downstream impact: connection-pool exhaustion in our app (timeout exceeded when trying to connect in pg-pool), and could not send data to client: Broken pipe on the server side, because queries block on disk and never release connections.
Volume ID / bind-mount: vol_ccbmenazoop2ufbv (bind-mounts/763a382a-4918-4e1c-87cc-36a169233029/...)
Please check I/O contention / storage health on the host backing this volume, and migrate the volume to healthy storage if needed. This is a production outage. Happy to provide full logs.
2 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Hi,
Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.
lawrencegripperwrk
Hi, Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.
a month ago
Thanks a lot for the quick turnaround, Lawrence — really appreciate you tracking down the network link.
I'm seeing improvement: a dd ... oflag=dsync write test went from ~36 kB/s (during the incident) to ~568 kB/s now. That said, it's still well below what I'd expect from healthy storage, so I don't think it's fully back to normal yet. Could you keep an eye on it / confirm whether more recovery is expected as things settle?
I'll keep monitoring on my side and follow up if it degrades again. Thanks again!