21 days ago
PostgreSQL 18.4 (image ghcr.io/railwayapp-templates/postgres-ssl) on a
persistent volume. Tiny, nearly-empty DB (~406MB), almost no traffic, one PG
service + one Next.js service on private networking.
A single-row UPDATE (by PK) intermittently takes 10–45s. App-side timing shows
select=5ms, bcrypt=240ms, update=45102ms — only the write blocks. Every slow
UPDATE coincides EXACTLY with a Postgres checkpoint whose FSYNC phase is huge
despite almost no data:
checkpoint complete: wrote 2 buffers (0.0%), ... write=0.226 s, sync=42.515 s,
total=44.723 s; sync files=3, longest=41.966 s; distance=8 kB
So write=0.2s but fsync=42s on 2 buffers. This has happened for ~1 month, even
after >24h of full idle, with no concurrent users and no heavy writes. PG config
is stock/reasonable (checkpoint_completion_target=0.9, checkpoint_timeout=300,
synchronous_commit=on, fsync=on). CPU/RAM idle, disk flat at ~406MB.
This looks like storage/volume fsync latency, not our SQL or PG config.
Questions:
-
Is this volume/host showing abnormal fsync/storage latency during
checkpoints? Any noisy-neighbor or backend I/O contention?
-
Is there known maintenance/rebalancing affecting this volume?
-
Can the volume be moved to a different backend/region?
I do NOT want to disable synchronous_commit/fsync (durability). Looking for the
root storage cause.
5 Replies
21 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 21 days ago
21 days ago
This is not a community-solvable issue — it's Railway storage-host I/O
degradation, already root-caused and FIXED by Railway staff in an identical
thread: "Severe Postgres disk I/O latency — 38s COMMIT on a 40MB database"
(bondi-admin). There, @chandrika (Railway) confirmed the host had degraded I/O
and STAGED A MIGRATION to a healthy host (apply via redeploy).
I'm on the exact same case: tiny DB, fsync=42s on 2 buffers. Please can a Railway
engineer check my Postgres volume's storage host and stage the same migration to
a healthy host? A community bounty can't resolve host-level storage.
21 days ago
Run it
21 days ago
10-4
21 days ago
Thanks, but this needs a Railway engineer (EMPLOYEE), not a community fix — the
resolution in the identical solved thread was a host migration staged by Railway
staff (@chandrika), which only Railway can do. Could a Railway engineer please
check my Postgres volume's storage host? Happy to provide project/service IDs.
18 days ago
Following up on this in case it was missed over the weekend.
This appears to match the previously resolved case where Railway identified a degraded storage host and staged a Postgres volume migration:
"Severe Postgres disk I/O latency — 38s COMMIT on a 40MB database."
My symptoms are effectively the same: tiny database, intermittent 10–45s transaction stalls, and PostgreSQL checkpoints showing 40+ second fsync times while writing only a couple of buffers.
Could a Railway engineer please check whether my Postgres volume is on a degraded storage host and, if necessary, stage a migration? I'm happy to provide the project ID, service ID, or any additional logs that would help.
Thanks!