calm-blue
burni80
FREEOP

8 days ago

Description: Self-hosted CouchDB for Obsidian LiveSync

Category: Storage

URL: https://railway.com/deploy/calm-blue-1

1 Replies

burni80
FREEOP

2 hours ago

Recurring extreme COMMIT/fsync latency on Postgres (volume-backed) — up to 43s

Hi @feelnopain,

This is the exact same root cause that another Railway user hit and had resolved back in May 2026. Your symptoms — multi-second bare COMMIT stalls, synchronous_commit=on, no replica, no application-level lock contention — are identical to that case, which Railway traced to a degraded storage host under the Postgres volume.

Railway's own conclusion (from employee chandrika, May 2026):

"Your Postgres instance is on a storage host that's experiencing degraded I/O performance. We've staged a migration to a newer, healthier host. Trigger a redeploy and the volume will migrate — storage latency should return to normal."

Solved thread: https://station.railway.com/questions/severe-postgres-disk-i-o-latency-38s-c-22bb5414


Immediate fix (stop the bleeding now — 1 minute, no downtime)

Set synchronous_commit to off so COMMITs stop waiting on the slow disk:

ALTER SYSTEM SET synchronous_commit = off;
SELECT pg_reload_conf();

What this does: WAL writes still happen, but COMMIT returns immediately before the fsync completes. In the event of a crash, you could lose the last transaction (typically <1 second of data). PostgreSQL itself is crash-safe — WAL integrity is preserved, only the timing of the flush changes. This is a widely-used setting in production. Flip it back to on once the storage host is migrated.

SHOW synchronous_commit;
-- should return: off

Permanent fix (needs Railway support)

Ask Railway to migrate your Postgres volume to a healthy storage host. Include:

Railway's chandrika confirmed the process: they stage the migration, you trigger a redeploy, the volume migrates along with the service. A few minutes of downtime.


Temporary workaround (no support needed)

Until Railway migrates your volume, redeploying to a different region can land you on a healthier storage host. Railway's ray-chen recommended this in the May case. Note: database downtime while the volume syncs to the new region.


Proactive monitoring (catch it early next time)

Run these periodically to catch fsync stalls before users do:

-- Recent slow statements (>1s)
SELECT
  query,
  calls,
  mean_exec_time::numeric(10,2) AS avg_ms,
  max_exec_time::numeric(10,2) AS max_ms
FROM pg_stat_statements
WHERE max_exec_time > 1000
ORDER BY max_exec_time DESC
LIMIT 10;

-- Checkpoint stats (high sync_time = storage issue)
SELECT
  checkpoints_timed,
  checkpoints_req,
  checkpoint_write_time::numeric(10,2) AS write_s,
  checkpoint_sync_time::numeric(10,2) AS sync_s
FROM pg_stat_bgwriter;

If checkpoint_sync_time spikes, or max_exec_time on trivial queries jumps to 10s+, your storage host is degrading again.


TL;DR

Right now: ALTER SYSTEM SET synchronous_commit = off; SELECT pg_reload_conf();

Then: Ask Railway to migrate your volume to a healthy host (same fix as May 2026).

While waiting: Redeploy to a different region as temporary relief.


Disclaimer: I don't work for Railway — just cross-referencing a previously resolved case with the same root cause.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...