Hi Railway, Pro workspace **Paris Group Projects** (`aa03ea5a-afa3-47b2-8acd-72061ed4f540`). Production Postgres on project **exact-aco** has been stalling on disk fsync since 16 Sep 2026. The factory UI freezes; statements over 20s are cancelled. It got worse today (17 Sep). This matches the known metal/host I/O pattern (checkpoint `sync=` spikes that do not track buffers written). Please inspect the volume host and move this volume/Postgres to a healthy host. **Do not restart or migrate during factory hours** without coordinating with us. ## Please do 1. Inspect disk / storage host for this volume instance and migrate off the noisy/degraded host if needed. 2. Confirm whether any platform snapshot/backup job besides the volume schedules below ran at the stall times. 3. If a restart or volume move is required, schedule it **before 10:00 UTC or after 21:30 UTC** (factory window is 07:00–18:30 America/Sao_Paulo, UTC−3). ## Resources - Workspace: Paris Group Projects `aa03ea5a-afa3-47b2-8acd-72061ed4f540` - Project: exact-aco `922d18a8-b326-4934-a392-78c3b50532f3` - Environment: production `fc07133b-dcfa-439c-8ab7-27ff48b3d324` - Service: Postgres `066f1f00-c2f3-418c-a153-092e54cc5e07` - Service instance: `d4a1f806-543e-4b47-accc-1fa870f253e1` - Deployment (unchanged since 2026-08-30): `5cc892fd-d3c3-4a82-82f0-39ac170b53ea` SUCCESS - Volume: postgres-volume `e6782dc9-f9ad-49a9-b087-e92b79bf3002` - Volume instance: `714f8451-c1fb-4b73-8320-a32dc826e04c` - Region: **us-east4-eqdc4a** - Image: `ghcr.io/railwayapp-templates/postgres-ssl:18` (PostgreSQL 18.6) - HA: standalone (not a cluster) - PITR: **disabled** (`enabled=false`, `bucketWired=false`) ## Why this looks like host disk, not our query load Railway metrics last 48h on this Postgres service: - CPU average 0.008 vCPU / limit 32 (≈0%) - Memory current ~515 MB / limit 32768 MB (1.6%) - Volume current ~8.9 GB / 50 GB (~18%) - Connections reported by the customer at the time: 13 of 500, no lock waits Checkpoint `sync=` does **not** track bytes written: ``` 2026-09-17 12:04:48 UTC wrote 573 buffers (3.5%) sync=42.729 s longest=24.518 s total=109.393 s 2026-09-17 10:49:01 UTC wrote 242 buffers (1.5%) sync=39.116 s longest=22.854 s 2026-09-17 03:48:29 UTC wrote 67 buffers (0.4%) sync=20.191 s longest=19.837 s 2026-09-17 01:13:32 UTC wrote 184 buffers (1.1%) sync=22.514 s longest=21.703 s 2026-09-17 01:12:21 UTC wrote 2969 buffers (18.1%) sync= 1.200 s longest= 0.735 s ``` At 12:04 UTC the same second as the 42.7s sync, Postgres cancelled statements (`ERROR: canceling statement due to statement timeout`) including a `select` on shipment/load data — that is the factory freeze. Baseline before 13 Sep: checkpoint sync ≤ 0.03 s. ## Volume backup schedules (we already checked) These are volume snapshots, not PITR: - Daily daily at 08:42 UTC → 08:42 UTC (05:42 America/Sao_Paulo). Last: 2026-09-17T08:42:06Z - Weekly weekly Saturday 04:21 UTC → Saturday 04:21 UTC - Monthly monthly 1st 20:10 UTC → 1st of month 20:10 UTC The stall times (BRT 00:48, 07:49, 08:49, 09:04, and 16 Sep 20:08 / 22:13) **do not line up** with the daily snapshot. Please confirm nothing else on the host was snapshotting at those times. Related public threads with the same shape: - https://station.railway.com/questions/database-performance-issue-more-slow-s-f9c7893e - https://station.railway.com/questions/severe-postgres-disk-i-o-latency-38s-c-22bb5414 Contact for maintenance window: Gustavo Paris (`gustavo.b.paris@gmail.com`, workspace Admin) and Kleyton Luza (Exact Aço, customer on this project). Thank you, Gustavo Paris Paris Group

Postgres volume I/O stall on us-east4-eqdc4a — checkpoint sync 42.7s, factory statement timeouts (exact-aco production)

gustavo-paris

PROOP

3 days ago

Hi Railway,

Pro workspace Paris Group Projects (aa03ea5a-afa3-47b2-8acd-72061ed4f540). Production Postgres on project exact-aco has been stalling on disk fsync since 16 Sep 2026. The factory UI freezes; statements over 20s are cancelled. It got worse today (17 Sep).

This matches the known metal/host I/O pattern (checkpoint sync= spikes that do not track buffers written). Please inspect the volume host and move this volume/Postgres to a healthy host. Do not restart or migrate during factory hours without coordinating with us.

Please do

Inspect disk / storage host for this volume instance and migrate off the noisy/degraded host if needed.
Confirm whether any platform snapshot/backup job besides the volume schedules below ran at the stall times.
If a restart or volume move is required, schedule it before 10:00 UTC or after 21:30 UTC (factory window is 07:00–18:30 America/Sao_Paulo, UTC−3).

Resources

Workspace: Paris Group Projects aa03ea5a-afa3-47b2-8acd-72061ed4f540
Project: exact-aco 922d18a8-b326-4934-a392-78c3b50532f3
Environment: production fc07133b-dcfa-439c-8ab7-27ff48b3d324
Service: Postgres 066f1f00-c2f3-418c-a153-092e54cc5e07
Service instance: d4a1f806-543e-4b47-accc-1fa870f253e1
Deployment (unchanged since 2026-08-30): 5cc892fd-d3c3-4a82-82f0-39ac170b53ea SUCCESS
Volume: postgres-volume e6782dc9-f9ad-49a9-b087-e92b79bf3002
Volume instance: 714f8451-c1fb-4b73-8320-a32dc826e04c
Region: us-east4-eqdc4a
Image: ghcr.io/railwayapp-templates/postgres-ssl:18 (PostgreSQL 18.6)
HA: standalone (not a cluster)
PITR: disabled (enabled=false, bucketWired=false)

Why this looks like host disk, not our query load

Railway metrics last 48h on this Postgres service:

CPU average 0.008 vCPU / limit 32 (≈0%)
Memory current ~515 MB / limit 32768 MB (1.6%)
Volume current ~8.9 GB / 50 GB (~18%)
Connections reported by the customer at the time: 13 of 500, no lock waits

Checkpoint sync= does not track bytes written:

2026-09-17 12:04:48 UTC  wrote 573 buffers (3.5%)   sync=42.729 s  longest=24.518 s  total=109.393 s
2026-09-17 10:49:01 UTC  wrote 242 buffers (1.5%)   sync=39.116 s  longest=22.854 s
2026-09-17 03:48:29 UTC  wrote  67 buffers (0.4%)   sync=20.191 s  longest=19.837 s
2026-09-17 01:13:32 UTC  wrote 184 buffers (1.1%)   sync=22.514 s  longest=21.703 s
2026-09-17 01:12:21 UTC  wrote 2969 buffers (18.1%) sync= 1.200 s  longest= 0.735 s

At 12:04 UTC the same second as the 42.7s sync, Postgres cancelled statements (ERROR: canceling statement due to statement timeout) including a select on shipment/load data — that is the factory freeze.

Baseline before 13 Sep: checkpoint sync ≤ 0.03 s.

Volume backup schedules (we already checked)

These are volume snapshots, not PITR:

Daily daily at 08:42 UTC → 08:42 UTC (05:42 America/Sao_Paulo). Last: 2026-09-17T08:42:06Z
Weekly weekly Saturday 04:21 UTC → Saturday 04:21 UTC
Monthly monthly 1st 20:10 UTC → 1st of month 20:10 UTC

The stall times (BRT 00:48, 07:49, 08:49, 09:04, and 16 Sep 20:08 / 22:13) do not line up with the daily snapshot. Please confirm nothing else on the host was snapshotting at those times.

Related public threads with the same shape:

Contact for maintenance window: Gustavo Paris (gustavo.b.paris@gmail.com, workspace Admin) and Kleyton Luza (Exact Aço, customer on this project).

Thank you,

Gustavo Paris

Paris Group

$20 Bounty

3 Replies

Railway

BOT

3 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 3 days ago

gustavo-paris

PROOP

2 days ago

Thanks @nnperic — we will take the operational point.

For Railway staff:

Please stage a volume/host migration whenever you are ready. We will trigger the redeploy ourselves in the factory-safe window: before 10:00 UTC or after 21:30 UTC (live factory is 07:00–18:30 America/Sao_Paulo). We will not ask you to wait on a call.

The number that carries the case: longest=24.518 s — a single fsync on one file. Baseline until 13 Sep was ≤ 0.03 s. Deployment unchanged since 2026-08-30 (5cc892fd). CPU 0.008/32, RAM ~515 MB / 32 GB, disk 8.9/50 GB.

Volume instance: 714f8451-c1fb-4b73-8320-a32dc826e04c
Region: us-east4-eqdc4a
Service: Postgres 066f1f00-c2f3-418c-a153-092e54cc5e07

We are not changing region or synchronous_commit during the shift. If you stage the migration today, we redeploy after 21:30 UTC.

Please confirm when the migration is staged on the service.

samikhkankhan

FREETop 5% Contributor

a day ago

The evidence points to storage/host I/O latency rather than query load.

The strongest indicator is the PostgreSQL checkpoint:

sync=42.729 s and longest=24.518 s

while only 573 buffers were written. PostgreSQL checkpoints flush dirty data to disk, so a 24+ second individual sync with CPU around 0.008/32 and only ~18% volume utilization is not explained by CPU, memory, or a full volume. The fact that the deployment has been unchanged since 2026-08-30 and the checkpoint latency was previously <=0.03s further supports an infrastructure/storage regression.

I would not change synchronous_commit or fsync as a workaround because that changes durability guarantees. PostgreSQL documents that fsync exists specifically to ensure durable recovery after a crash.

The correct remediation is:

Have Railway inspect the underlying storage/host for volume instance 714f8451-c1fb-4b73-8320-a32dc826e04c.
Compare host/storage I/O latency with other Postgres volumes in us-east4-eqdc4a.
Check host-level snapshot, storage, or maintenance activity around the exact 20–40s fsync events.
If the host/storage is degraded, migrate the volume/service to a healthy host.
Perform the migration only inside the customer's stated maintenance window.
After migration, verify that checkpoint sync/longest return near the previous baseline and that the 20s statement timeouts disappear.

Railway already documents that host faults can trigger migrations, and that volume migration can cause downtime, so the requested controlled migration is appropriate here.

I would also capture one before/after checkpoint sample. If the same workload produces milliseconds rather than tens of seconds after the host move, that provides a very strong confirmation that the underlying storage host was the cause.

No application query rewrite or database tuning should be the first response to this evidence. The immediate issue is the abnormal disk synchronization latency.

gustavo-paris

PROOP

3 hours ago

Railway staff — ping.

This Pro thread has been open ~61 hours with no engineer reply. Checkpoint longest=24.518 s is still the case. Factory is currently off (02:43 America/Sao_Paulo); we can redeploy until 10:00 UTC today.

Please stage the volume/host migration on Postgres 066f1f00-c2f3-418c-a153-092e54cc5e07 / volume instance 714f8451-c1fb-4b73-8320-a32dc826e04c (us-east4-eqdc4a). We will trigger the redeploy as soon as it is staged, inside that window.

If you cannot stage before 10:00 UTC, we will redeploy after 21:30 UTC.

Welcome!