We run a small Postgres 17 database on a managed volume. It is well under a gigabyte, sits almost entirely in page cache (99.8% buffer hit ratio), and writes roughly 1.5 kB/s of WAL. For about a month it has had multi-second write stalls that we cannot explain from the workload, and I would like help identifying the storage-side condition behind them.

WHAT WE SEE

An isolated COMMIT stalling on an idle service. In one 10-minute window our client instrumentation recorded ~900 database operations, of which EXACTLY ONE exceeded 10 seconds - and nothing at all fell between 1 s and 10 s. Broken down by operation type, the entire outlier was COMMIT: zero on SELECT, INSERT, UPDATE, DELETE or BEGIN. The service was doing well under one request per second at the time. No locks, no CPU pressure, no queueing.

It is chronic. Roughly 1% of ALL commits exceed 5 seconds, every day, across 14 days of telemetry, while the median commit is 3.3 ms.

Checkpoint fsync from the Postgres log, with log_checkpoints on. The clearest single line:

checkpoint complete: wrote 12 buffers (0.1%); write=1.214 s, sync=10.521 s, total=11.752 s; sync files=10, longest=10.505 s; distance=49 kB

That is 96 kB taking 10.5 seconds to fsync. Checkpoints with a longest fsync of 5 s or more went from about 1/day to about 4.7/day over three weeks, p99 16.9 s, max 29.1 s.

THE MEASUREMENT I MOST WANT EXPLAINED

Read from inside the container, /proc/diskstats lists 1,830 zvol devices on this host. Of the 516 with at least 50,000 lifetime writes:

• median write latency across those volumes: 21 ms
• our volume: 62.3 ms per write (2.8 million writes), 13.0 ms per flush
• some volumes on the same host: under 1 ms per write

So volumes on one host differ by nearly two orders of magnitude. /proc/pressure/io on that host shows full avg300 of 16-22% while our database is idle, and cumulative "full" time is about 8.6% of the host's ~200-day uptime. Our container's own limits are nowhere near reached (io.max 70 MB/s and 3,000 IOPS; cpu.stat nr_throttled 0; memory pressure ~0; 245 MB used of a 7 GB limit).

READS BLOCK THE SAME WAY, which we think rules out lock contention: with track_io_timing on, individual SELECTs on small tables took up to 18.4 s with 99.9% of that time inside block reads, and our average block-read time is 12.1 ms despite the 99.8% cache hit ratio. A lock cannot block a read syscall.

A SECOND DATABASE OF OURS, ON A DIFFERENT HOST, stalled in the same minutes as this one on four separate occasions. Both database hosts show io pressure full avg300 around 16-17%; our two application hosts show under 2%.

ONE THING WE HAVE RULED OUT AS A REMEDY: every restart is followed within 5-45 minutes by a burst of reads over 5 seconds (17 in one minute after one restart). On four consecutive days with no restart there were ZERO reads over 5 s out of 505,000. A cold page cache on a device averaging 12 ms per block read is its own outage.

QUESTIONS

1. What makes one zvol on a host average 62 ms per write while the median of its neighbours is 21 ms and some are under 1 ms? Different backing tier, a degraded vdev, ZFS fragmentation, a noisy tenant on the same pool?
2. Is there anything readable from inside a container that would distinguish those causes? We have /proc/diskstats, /proc/pressure/io and the cgroup io.* files.
3. Is a 10-second fsync of 96 kB, on an idle database, ever explainable by anything other than the storage layer?

Happy to share the full parsed checkpoint history (2,092 checkpoints), relation sizes, or raw logs.