Update: my diagnosis was wrong. This is not a disk I/O problem, and no volume migration is needed. Marking this as resolved — please disregard the original request.

Thank you both for taking the time to look at this — you gave careful, well-structured answers and they deserved a better question. The evidence I posted doesn't survive scrutiny, and I'd rather correct the record publicly than have this thread mislead the next person who searches for it.

What I got wrong. I read checkpoint complete: write=25s for ~2MB as "the disk is writing at <100 KB/s". But write= doesn't measure disk throughput. Every checkpoint on this instance is checkpoint starting: time — deliberately spread out by checkpoint_completion_target. While the checkpointer is ahead of schedule it sleeps 100ms after each buffer (pg_usleep(100000L) in CheckpointWriteDelay). On a 677MB database it is always ahead of schedule, so write time is simply buffers × 100ms.

I went back and measured write / buffers across 40+ checkpoints from Jul 5 to Jul 17, with buffer counts from 1 to 463:

| when | buffers | write= | ms/buffer |

|---|---|---|---|

| Jul 16 22:07 ("healthy") | 1 | 0.118s | 118 |

| Jul 16 23:43 ("degraded") | 280 | 28.4s | 101.4 |

| Jul 17 00:18 ("degraded") | 453 | 45.4s | 100.2 |

| Jul 17 12:12 (now) | 250 | 25.1s | 100.2 |

It is ~100ms/buffer everywhere. My "healthy 0.2s checkpoints" were 1-buffer checkpoints on an idle database; the "degraded 25-105s" ones were 250-450 buffers under real load. Identical rate. The healthy-vs-degraded comparison — singled out as the strongest evidence — was just idle vs. busy.

Two further corrections to my own report:

• The post-restart write=0.156s I offered as proof that "the disk recovered" was an end-of-recovery immediate checkpoint. Immediate checkpoints skip the pacing entirely, so it was never comparable to the paced ones.
• The restart didn't fix anything. I still got canceling statement due to statement timeout on a trivial INSERT at 01:08:42 UTC — 22 minutes after the database came back at 00:46:50, with the 00:57 checkpoint pacing normally at sync=0.019s. The symptom cleared on its own; the restart just happened to be nearby.

What actually survives. The timeouts were real, and a few signals do hold up: two checkpoints at ~250-300 ms/buffer (so ~150-200ms of genuine extra latency per buffer), one fsync at 5.9s, and several checkpoints where total far exceeds write+sync. But that's a handful of outliers over two weeks — not a degraded volume — and the instance is clean on every one of those measures today.

My working hypothesis now is CPU steal / a busy host rather than storage: it fits a 100ms sleep stretching to 250ms, and a PK SELECT taking 30s while the container's own CPU sits at 0.1% (the container is idle — something else isn't). The other candidate I haven't ruled out is an ACCESS EXCLUSIVE lock from a migration that deployed that night; my pg_stat_activity check ran at 01:04 and the timeout hit at 01:08, so it missed the window entirely. I've enabled log_lock_waits and log_min_duration_statement=5s, so if it recurs I'll have direct evidence instead of inference.

Re: the dump/restore-and-compare and region-swap suggestions — thank you, both are sensible answers to the question I originally asked. But given the correction above, I won't do either: the symptom is episodic and the database currently measures healthy on every axis, so a fresh volume would benchmark identically to the current one (nothing to compare against), and the region-swap costs two voluntary downtime windows to treat a cause I can no longer support with evidence. If the real trigger is host-level or a lock on my side, it would follow me to the new volume anyway. The dump-to-new-service comparison is a good playbook for sustained DB-side degradation, and I'm keeping it for that case.

So: no on-call escalation, no volume migration, closing this out. Sorry for the noise, and thanks again.

If there's one thing worth leaving here for the next person: checkpoint write= and total= are not disk throughput measurements. Normalize by buffer count before you conclude anything. A paced checkpoint on a small database will sit at ~100ms/buffer no matter how fast your disk is.