Thanks @nnperic — we will take the operational point.

For Railway staff:

Please stage a volume/host migration whenever you are ready. We will trigger the redeploy ourselves in the factory-safe window: before 10:00 UTC or after 21:30 UTC (live factory is 07:00–18:30 America/Sao_Paulo). We will not ask you to wait on a call.

The number that carries the case: longest=24.518 s — a single fsync on one file. Baseline until 13 Sep was ≤ 0.03 s. Deployment unchanged since 2026-08-30 (5cc892fd). CPU 0.008/32, RAM ~515 MB / 32 GB, disk 8.9/50 GB.

• Volume instance: 714f8451-c1fb-4b73-8320-a32dc826e04c
• Region: us-east4-eqdc4a
• Service: Postgres 066f1f00-c2f3-418c-a153-092e54cc5e07

We are not changing region or synchronous_commit during the shift. If you stage the migration today, we redeploy after 21:30 UTC.

Please confirm when the migration is staged on the service.

Railway staff — ping.

This Pro thread has been open ~61 hours with no engineer reply. Checkpoint longest=24.518 s is still the case. Factory is currently off (02:43 America/Sao_Paulo); we can redeploy until 10:00 UTC today.

Please stage the volume/host migration on Postgres 066f1f00-c2f3-418c-a153-092e54cc5e07 / volume instance 714f8451-c1fb-4b73-8320-a32dc826e04c (us-east4-eqdc4a). We will trigger the redeploy as soon as it is staged, inside that window.

If you cannot stage before 10:00 UTC, we will redeploy after 21:30 UTC.

Railway staff — 72h Pro SLA is about to lapse (opened 2026-09-17 16:24 UTC; ~40 minutes left) with no engineer reply.

Production Postgres on exact-aco is still on volume instance 714f8451-c1fb-4b73-8320-a32dc826e04c (us-east4-eqdc4a). Checkpoint longest=24.518 s. Factory is live until 21:30 UTC; we will redeploy after that if you stage the host/volume migration.

Please stage it. We will pull the trigger after 21:30 UTC today.

Update from us (Sunday factory-off window):

We tried to self-migrate the volume off the noisy host.

1. Locked volume backup pre-metal-exit-2026-09-20 (a88a34ac-5700-4ee3-adcd-70b3763a94f5).
2. Set service region to us-east4 (GCP) — Railway remapped it to us-east4-eqdc4a (deprecatedRegions).
3. Set us-east4-eqdc16a — deploy still landed on us-east4-eqdc4a.
4. Set us-west2 — same: volume instance 714f8451-c1fb-4b73-8320-a32dc826e04c stays us-east4-eqdc4a READY. Environment config volumes.*.region is not writable.

The volume is pinned to this Metal host. Customer-side region change cannot move it. Please migrate this volume to a healthy storage host (or unstick the region pin). We can redeploy immediately once it is staged.

Postgres is back up after the redeploys (8c2cbded, SUCCESS). Factory-safe window continues until 10:00 UTC.

Fresh checkpoint evidence after tonight's redeploys (volume still pinned on us-east4-eqdc4a):

2026-09-20 21:45:22 UTC  wrote 2 buffers    sync=0.006 s  longest=0.004 s
2026-09-20 21:55:37 UTC  wrote 140 buffers  sync=0.005 s  longest=0.005 s
2026-09-20 22:05:34 UTC  wrote 102 buffers  sync=0.532 s  longest=0.524 s

Not the 42 s factory freeze yet, but longest jumped two orders of magnitude on a small write (102 buffers) ten minutes after a healthy 140-buffer checkpoint. Same host, same volume 714f8451-c1fb-4b73-8320-a32dc826e04c. Please migrate the volume off this storage host. We can redeploy immediately (factory is off until 10:00 UTC).

The stall is back on the same pinned volume (714f8451-c1fb-4b73-8320-a32dc826e04c, us-east4-eqdc4a), factory currently off:

2026-09-21 01:17:58 UTC  wrote 1276 buffers (7.8%)  write=138.9 s  sync=12.587 s  longest=8.482 s

Same shape as 17 Sep (longest=24.5 s). Ten minutes earlier a 201-buffer checkpoint was longest=0.005 s. Please migrate this volume off the host. We can redeploy until 10:00 UTC.

Follow-up: ERROR: canceling statement due to statement timeout at 2026-09-21 01:37:23 UTC on a romaneio_prancha / romaneio join (factory shipment screen).

Checkpoints immediately around it were healthy (longest=0.008 s at 01:35). The 12.6 s fsync was 20 minutes earlier (01:17). Same volume 714f8451-…. Reads are stalling too, not only checkpoint sync.

Second factory query timeout: 2026-09-21 01:44:13 UTC on pedido list (id, numero, descricao, construtora_id, …). Previous was 01:37:23 on romaneio_prancha. Checkpoint at 01:40:33 was longest=0.015 s. Reads stalling with healthy fsync — same pinned volume.

Third factory timeout: 2026-09-21 01:47:50 UTC on romaneio_item ⋈ alocacao.

Tiny checkpoint two minutes earlier — this is the original smoking gun (bytes written do not explain fsync):

2026-09-21 01:45:44 UTC  wrote 5 buffers (0.0%)  sync=0.974 s  longest=0.968 s

Three statement timeouts in 10 minutes (01:37 romaneio, 01:44 pedido, 01:47 alocacao). Volume still 714f8451-c1fb-4b73-8320-a32dc826e04c / us-east4-eqdc4a.

Factory is live (America/Sao_Paulo 08:37). Same pinned volume, stall during shift:

2026-09-21 11:37:59 UTC  wrote 1355 buffers (8.3%)  sync=7.256 s  longest=7.249 s

We will not restart or migrate during 10:00–21:30 UTC. Please stage the volume move; we redeploy after 21:30 UTC.

Second stall this shift, 14 min later:

2026-09-21 11:51:58 UTC  wrote 411 buffers (2.5%)  sync=3.163 s  longest=2.940 s

Still below the app 20 s timeout. We are not restarting during factory hours.

Factory freeze during shift (America/Sao_Paulo 08:55):

ERROR: canceling statement due to statement timeout on

UPDATE exact_aco.alocacao SET maquina_id=… (shop-floor machine allocation).

Three minutes after:

2026-09-21 11:51:58 UTC  wrote 411 buffers  sync=3.163 s  longest=2.940 s

We are not restarting. Volume still 714f8451-c1fb-4b73-8320-a32dc826e04c / us-east4-eqdc4a.

Second factory timeout this hour: 2026-09-21 12:00:58 UTC on SELECT romaneio_item_id FROM apontamento WHERE romaneio_item_id IN (…) (shop-floor production notes). Previous was 11:55:06 UPDATE alocacao. Still not restarting during shift.

Third factory timeout this hour: 2026-09-21 12:02:31 UTC on SELECT … FROM chamado (support tickets). Checkpoint 50 s earlier was longest=0.005 s — reads stalling with healthy fsync. Not restarting.

Another stall during shift:

2026-09-21 14:11:08 UTC  wrote 250 buffers (1.5%)  sync=5.112 s  longest=5.089 s

Five minutes earlier the same size class was longest=0.026 s. Not restarting.

Still stalling during the shift. Cutover to a new volume is scheduled for 04:30–08:00 UTC (01:30–05:00 America/Sao_Paulo). We will not restart this instance before then.

2026-09-21 16:16:20 UTC  wrote 358 buffers (2.2%)  sync=2.274 s  longest=2.230 s

Factory is live. Worst stall since 17 Sep, same pinned volume:

2026-09-21 19:20:48 UTC  wrote 2372 buffers (14.5%)  sync=31.006 s  longest=16.637 s

Same second: canceling statement due to statement timeout while locking apontamento (SELECT … FOR UPDATE on open shop-floor notes).

We are not restarting. Customer cutover to a new volume is scheduled for 04:30 UTC (01:30 America/Sao_Paulo).

Factory is now off. Same smoking gun, small write, long fsync:

2026-09-22 00:26:47 UTC  wrote 212 buffers (1.3%)  sync=18.167 s  longest=18.157 s

Customer cutover (new volume, fresh dump) starts at 04:30 UTC / 01:30 America/Sao_Paulo. This instance stays up until then.

sekitar 61 jam tanpa ada balasan dari teknisi. Checkpoint longest=24.518 smasih berlaku. Pabrik saat ini sedang tidak beroperasi (02:43 America/Sao_Paulo); kita dapat melakukan penyebaran ulang hingga pukul 10:00 UTC hari ini .

Harap siapkan migrasi volume/host pada 066f1f00-c2f3-418c-a153-092e54cc5e07instance Postgres/volume 714f8451-c1fb-4b73-8320-a32dc826e04c( us-east4-eqdc4a). Kami akan memicu penyebaran ulang segera setelah disiapkan, dalam jangka waktu tersebut.

Jika Anda tidak dapat melakukan staging sebelum pukul 10:00 UTC, kami akan melakukan redeploy setelah pukul 21:30 UTC