Postgres volume 52b76bee freezes all I/O for 30-120s intermittently (IO:WalSync / WAL fsync stalls)
fieldkiller
PROOP

a month ago

Volume 52b76bee-d48d-4bde-a69a-0a6b688f8a8f (postgres-volume, attached to Postgres-L0gI, mount /var/lib/postgresql/data; region us-east4 per a prior probe, please confirm) intermittently STOPS acknowledging all writes for 30-120s, then recovers cleanly, several times per hour since ~2026-07-15. Disk is 8GB/50GB (13%).

During each freeze every physical write blocks: backends sit in IO:WalSync / LWLock:WALWrite (WAL fsync) and IO:DataFileExtend. CPU is idle (~0.3% of quota), memory flat. It is NOT load-induced (the lowest-traffic hour of the day has the most freezes) and NOT our Postgres config.

DIRECT EVIDENCE (dual-host, same region, 2026-07-21). A server-side commit-fsync probe (pg_logical_emit_message(true,...) + COMMIT) run in parallel on Postgres-L0gI (prod) AND a second idle Postgres in the same region:

  • 16:34:27-16:35:19 UTC: prod COMMIT hung 52.5s on IO:WalSync; the idle sibling ran ~48ms/commit, never blocked.
    • 16:35:52-16:37:11 UTC: prod COMMIT hung 79.0s; sibling still ~48ms.
    • Over that window prod's max commit batch was 78,962ms vs the sibling's max 157ms -- a ~500x tail gap on the identical operation at the same instants.
  • The healthy same-region sibling rules out a region-wide cause; the IO:WalSync wait (a kernel fdatasync on the WAL file) places the stall below Postgres, at the block device.

BASELINE (pg_stat_io, track_io_timing on): even non-freeze WAL fsync averages ~36ms (walwriter 49.5ms) vs 0.1-0.5ms for a healthy NVMe; DataFileExtend 284.7ms/extend. The volume is chronically slow with a catastrophic tail.

IMPACT: intermittent site-wide 500s/timeouts (reads too, via connection-pool saturation), including delayed Stripe webhook processing.

RULED OUT on our side: CPU throttling (0.3% during a live freeze), memory, disk space, WAL/checkpoint pressure (1 requested vs 1151 timed checkpoints in 4 days, wal_buffers_full=0), synchronous replication (none configured), scheduled jobs / autovacuum / backup (freeze onsets don't align), deadlocks (0), and load.

ASKS:

  1. Do your storage-backend logs show maintenance, live-migration, replica/placement failover, or network events on the node backing volume 52b76bee-d48d-4bde-a69a-0a6b688f8a8f at the UTC timestamps above (esp. 16:34:27 and 16:35:52 on 2026-07-21)?
  2. What are the IOPS / throughput / latency guarantees and any burst model for this volume? Is it on shared storage with noisy-neighbor exposure?
  3. Can you migrate this volume/service to a healthy node, and confirm whether the current node is degraded? Our same-region idle sibling performing 9.8x-500x better suggests this placement specifically is unhealthy.
$20 Bounty

1 Replies

Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 1 month ago


msns-dev
FREE

24 days ago

To permanently resolve the IO stalls and latency degradation, your volume must be shifted off the degraded hardware.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...