Skip to content
Production outage — severe disk I/O degradation on Postgres volume (DB up but unusable)
contas-ginga
OP

4 months ago

Our production Postgres service is experiencing severe, sustained disk I/O latency that makes the database effectively unusable. This is a platform/storage issue, not an application-level one — the database itself is healthy and reports ready to accept connections.

Evidence:

fsync of tiny amounts of data takes 10–18 seconds. Postgres checkpoints writing only ~100–700 kB routinely take 80–100 seconds total (sync phase alone 14–18s). Degrading over several days, became a full outage today.

The volume is at 1% usage (354 MB used of 46 GB) — not a capacity issue.

Small DB (~354 MB), minimal write load (checkpoint distance only a few hundred kB per cycle).

Live write test confirms it: dd if=/dev/zero of=/testfile bs=8k count=1000 oflag=dsync (8 MB) does not complete in over a minute. Healthy storage does this in a couple seconds.

We already redeployed the Postgres service. Compute moved to a new host, but I/O latency persists — isolating the problem to the storage volume itself, not the compute host.

Last boot's crash recovery logged syncing data directory (fsync), elapsed time: 11.59 s — abnormal for a tiny data dir.

Downstream impact: connection-pool exhaustion in our app (timeout exceeded when trying to connect in pg-pool), and could not send data to client: Broken pipe on the server side, because queries block on disk and never release connections.

Volume ID / bind-mount: vol_ccbmenazoop2ufbv (bind-mounts/763a382a-4918-4e1c-87cc-36a169233029/...)

Please check I/O contention / storage health on the host backing this volume, and migrate the volume to healthy storage if needed. This is a production outage. Happy to provide full logs.

$20 Bounty

3 Replies

Railway
BOT

4 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 4 months ago


lawrencegripperwrk
EMPLOYEE

4 months ago

Hi,

Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.


lawrencegripperwrk

Hi, Sorry you hit this issue, I identified a problematic network link and have remediated it. You should start to see recovery.

contas-ginga
OP

4 months ago

Thanks a lot for the quick turnaround, Lawrence — really appreciate you tracking down the network link.

I'm seeing improvement: a dd ... oflag=dsync write test went from ~36 kB/s (during the incident) to ~568 kB/s now. That said, it's still well below what I'd expect from healthy storage, so I don't think it's fully back to normal yet. Could you keep an eye on it / confirm whether more recovery is expected as things settle?

I'll keep monitoring on my side and follow up if it degrades again. Thanks again!


nazmulhaquebz

a month ago

Hello Railway Team,

This is a critical production outage affecting our PostgreSQL database. We have already redeployed the service and moved compute to a new host, but the severe disk I/O latency persists.

Our tests show that even an 8 MB synchronous write takes more than a minute, while small PostgreSQL checkpoints routinely take 80–100 seconds. The volume is only approximately 1% utilized, so capacity exhaustion does not appear to be the cause.

Could you please investigate the underlying storage health and I/O performance for volume vol_ccbmenazoop2ufbv and arrange a safe migration to healthy storage if necessary?

The issue is exhausting our application's database connection pool and disrupting production traffic. We would greatly appreciate urgent assistance and updates on the investigation.

We can provide the complete PostgreSQL logs and additional diagnostic evidence if required.

Thank you for your support.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...