a month ago
Volume 52b76bee-d48d-4bde-a69a-0a6b688f8a8f (postgres-volume, attached to Postgres-L0gI, mount /var/lib/postgresql/data; region us-east4 per a prior probe, please confirm) intermittently STOPS acknowledging all writes for 30-120s, then recovers cleanly, several times per hour since ~2026-07-15. Disk is 8GB/50GB (13%).
During each freeze every physical write blocks: backends sit in IO:WalSync / LWLock:WALWrite (WAL fsync) and IO:DataFileExtend. CPU is idle (~0.3% of quota), memory flat. It is NOT load-induced (the lowest-traffic hour of the day has the most freezes) and NOT our Postgres config.
DIRECT EVIDENCE (dual-host, same region, 2026-07-21). A server-side commit-fsync probe (pg_logical_emit_message(true,...) + COMMIT) run in parallel on Postgres-L0gI (prod) AND a second idle Postgres in the same region:
- 16:34:27-16:35:19 UTC: prod COMMIT hung 52.5s on IO:WalSync; the idle sibling ran ~48ms/commit, never blocked.
-
- 16:35:52-16:37:11 UTC: prod COMMIT hung 79.0s; sibling still ~48ms.
-
- Over that window prod's max commit batch was 78,962ms vs the sibling's max 157ms -- a ~500x tail gap on the identical operation at the same instants.
- The healthy same-region sibling rules out a region-wide cause; the IO:WalSync wait (a kernel fdatasync on the WAL file) places the stall below Postgres, at the block device.
BASELINE (pg_stat_io, track_io_timing on): even non-freeze WAL fsync averages ~36ms (walwriter 49.5ms) vs 0.1-0.5ms for a healthy NVMe; DataFileExtend 284.7ms/extend. The volume is chronically slow with a catastrophic tail.
IMPACT: intermittent site-wide 500s/timeouts (reads too, via connection-pool saturation), including delayed Stripe webhook processing.
RULED OUT on our side: CPU throttling (0.3% during a live freeze), memory, disk space, WAL/checkpoint pressure (1 requested vs 1151 timed checkpoints in 4 days, wal_buffers_full=0), synchronous replication (none configured), scheduled jobs / autovacuum / backup (freeze onsets don't align), deadlocks (0), and load.
ASKS:
- Do your storage-backend logs show maintenance, live-migration, replica/placement failover, or network events on the node backing volume 52b76bee-d48d-4bde-a69a-0a6b688f8a8f at the UTC timestamps above (esp. 16:34:27 and 16:35:52 on 2026-07-21)?
- What are the IOPS / throughput / latency guarantees and any burst model for this volume? Is it on shared storage with noisy-neighbor exposure?
- Can you migrate this volume/service to a healthy node, and confirm whether the current node is degraded? Our same-region idle sibling performing 9.8x-500x better suggests this placement specifically is unhealthy.
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
24 days ago
To permanently resolve the IO stalls and latency degradation, your volume must be shifted off the degraded hardware.