14 days ago
Project: 78136063-ddb8-4861-ae8f-dd4af962620b
Service: 320dfdfd-e98d-4b14-847e-5f7cabd062c7
Environment: production
Volume: d3b81e13-2f64-46b6-907d-ce3483bb19cd
mount /var/lib/postgresql/data, device /dev/zd5024
We're seeing otherwise-trivial requests stall for 30–90s. Investigation points to pathological fsync latency on the Postgres data volume, not load or our queries.
Evidence — pg_test_fsync run inside the Postgres container on the data volume:
Compare file sync methods using one 8kB write:
open_datasync 1.600 ops/sec 624,839 usecs/op
fdatasync 0.274 ops/sec 3,649,858 usecs/op (Postgres default)
fsync 0.738 ops/sec 1,355,646 usecs/opNon-sync'ed 8kB writes: 1,325,104 ops/sec (1 usec/op)
fdatasync is 0.27 ops/sec (~3.6s per fsync) — roughly 3–4 orders of magnitude slower than expected. Non-durable writes are instant (1.3M ops/sec), so the issue is isolated to durable sync/fsync on this volume.
Supporting context (all healthy, rules out our side):
DB size 109 MB; cache hit 99.99%; volume 1% used (165M/46G); no replication slots; checkpoints healthy (11,425 timed / 1 requested).
pg_stat_activity during incidents: with only ~3 active queries, 2 sit in wait_event = WALWrite/WALSync, query age climbing to 7–90s; wait_event_type='Lock' count = 0 (no lock contention).
CPU idle on both the DB and app services throughout.
Ask: Please investigate the I/O / fsync health of this volume and its backing host (appears to be a ZFS zvol). We suspect a degraded or misconfigured backing pool (e.g. ZIL/SLOG or a contended host). If the volume/host can't be remediated, please advise on relocating the service to healthy storage. The disk-usage metric for this volume also reports 0 MB/50000 MB, which seems related/abnormal.
4 Replies
Status changed to Awaiting Railway Response Railway • 14 days ago
14 days ago
This has merit. The volume is READY and not full from our side, but the Postgres runtime logs back up what you're seeing: checkpoint sync phases are repeatedly taking seconds, and there are abnormal checkpoint totals in the 40-80s range under very small write volume.
So I agree this points at the durable flush path for the volume/backing host rather than query load, CPU, or disk usage. I'm routing this for storage/host investigation. Please do not disable fsync in production; if we can't remediate the backing storage cleanly, the practical next step will be relocating the workload/volume to healthy storage.
Status changed to Awaiting User Response Railway • 14 days ago
13 days ago
Thanks, any chance you can switch synchronous_commit = off on that instance in the interim to alleviate the massive bottleneck slowing my app down?
Status changed to Awaiting Railway Response Railway • 13 days ago
13 days ago
Or don't I guess...
I tried to migrate the service to a different region to try get the volume onto a healthy disk but it got stuck at 94% with a very helpful "Failed to copy data" and "An error occurred during the data migration" error messages.
Ended up having to just deploy a fresh Postgres service and volume and migrate data across.
This is kind of the exact thing that you'd expect a platform like Railway to take off your plate to worry about.
13 days ago
Sorry about all of this. The migration failure on top of the storage issue made a bad situation worse, and you're right that this should have been smoother on our end. Your new Postgres service and volume are on healthy storage, so you should be in good shape going forward.
Status changed to Awaiting User Response Railway • 13 days ago
Status changed to Solved noahd • 13 days ago