2 days ago
Hi Railway team,
Our production Postgres volume is flushing to disk extremely slowly right now, and it's causing timeouts across our app. We need help urgently.
- Postgres service: 4e955526
- Volume: postgres-volume 658dc20e
- Postgres 17.11, container memory 24 GB
What we measure from inside Postgres:
- Right now (Oct 3, ~21:23 UTC) each WAL fsync takes ~1,046 ms (25 fsyncs in 30 s). Earlier today it was 562–672 ms. Our 8-day average is 11 ms; healthy is 1–2 ms.
- Our write load is tiny at the same time: WAL ~0.38 MB/s, background writer ~0.7 MB/s, checkpoint writes 0/s. The disk is flushing ~100x slower than normal with almost nothing to flush.
- Effect: every commit waits behind a ~1 s flush, 10–30 sessions wait on WAL at once, and our app requests time out.
- Started around 20:13 UTC today and is ongoing. Earlier stall windows today: 09:00–09:30 and 13:15–13:50 UTC.
- This looks like the storage-layer issue in thread "Severe storage I/O degradation on Postgres host" from Aug 29.
Questions:
- Is there a storage incident or a noisy neighbor on the host behind this volume?
- Are we being throttled, and what IOPS/throughput is this volume provisioned for?
- Can you fix it now (move the volume to a healthy host or a faster tier), and what would it cost?
We have not restarted the database and will not unless you advise it.
Thanks,
Trey, Carwash Blitz
3 Replies
Status changed to Awaiting Railway Response Railway • 2 days ago
2 days ago
Update + correction — it got worse. Correct volume ID is 43de19d9 (service 4e955526, region sfo); the 658dc20e in my first post was wrong.
Between ~23:16 and ~23:21 UTC today the database completed no commits at all for ~4–5 minutes: pg_current_wal_lsn did not move, pg_stat_wal.wal_sync did not increment, 27 sessions were waiting on WALWrite/WalSync (worst WAL wait 97 s), and the WAL writer itself was stuck in WALWrite. Reads kept working. At ~23:21 UTC writes resumed (32 syncs in 20 s) and the backlog drained.
Outside that freeze, fsync is still ~1 s per sync at under 1 MB/s writes. This is a production outage for our customers' texting. Can you please treat this as a storage incident and either move the volume to healthy storage or advise a migration path? We have not restarted anything.
2 days ago
URGENT UPDATE — ongoing production write outage. Please escalate this to the infrastructure/on-call team now.
The problem has worsened. Our latest full write freeze ran from 2026-10-04 00:18:22 to approximately 00:26 UTC: roughly eight minutes, zero commits, and 45 sessions waiting on WAL. After writes resumed, we measured 19.5 seconds per fsync while the backlog drained. Since then, fsync has remained 1.2–4.6 seconds.
Freeze windows across tonight (all UTC):
- 2026-10-03 23:16–23:21
- 2026-10-03 23:28–23:29
- 2026-10-03 23:54–23:56
- 2026-10-04 00:18:22–approximately 00:26
Write load is normal and low. Earlier measurements showed 1–4 seconds per fsync at only 100–200 IOPS. At 00:21 UTC during the latest freeze, we measured 15 IOPS and no fsync completions. Slowness does not track our application load. These are measurements from our database, not Railway's disk-performance metrics.
Identifiers, to clarify the earlier volume correction:
- Project: cwb-core — 04088fd8-9172-40b0-a981-29c13bf6a945
- Environment: production — 2121494b-908e-44ea-a8c1-a198509417b6
- Postgres service: 4e955526-f4f6-4659-9874-7a474069c7b6
- Attached logical volume: postgres-volume — 658dc20e-9393-4b4a-ae45-d90141489e70 (confirmed by Railway's current service description and dashboard)
- Region: sfo; capacity 500 GB, approximately 258 GB used
The earlier 43de19d9 prefix has not been mapped to a volume instance by the available tooling. Please identify the backing instance/host from the verified service and logical volume above.
Railway runtime logs for 23:50–00:35 UTC returned 204 statement timeouts and 56 lock timeouts. They include "checkpoint starting: time" at 2026-10-03 23:56:14.290 UTC; that filtered interval returned no checkpoint-complete line. The accessible Railway agent/API exposes disk space usage but no volume I/O latency, IOPS, queue or throttling telemetry.
Please treat this as an ongoing storage-path production outage. We need human infrastructure investigation of the volume/storage host, device/controller latency, queueing, throttling, host errors and maintenance during these windows, followed by urgent intervention or a coordinated, data-preserving move to healthy storage in sfo. Please confirm the actual IOPS/throughput policy, whether limits were hit, whether a same-region relocation is available, its expected downtime and any cost.
Please acknowledge infrastructure escalation and give the next action and ETA. We have performed no restart, redeploy or volume modification through this investigation.
Trey / Carwash Blitz
2 days ago
Confirming that we're seeing this on our side, have called an incident here and are looking into this: https://status.railway.com/incident/A0O39CGE
Redeploying or restarting will not help here, we suggest holding off on both for now
Our infrastructure team is looking into it, and we'll update the incident as soon as we have more
Status changed to Awaiting User Response Railway • 1 day ago