Postgres crash from ENOSPC + read-only filesystem remount (production, 2026-06-15)

Question

Hi folks, just experienced a crash that sounded like "no space on disk" while it was using just 315mb of allocated 5gb. I restarted the db and it got up and running. I checked the railway status and it didn't show any downtime, etc so sharing the details here (railway agent drafted me this):

**Severity:** High — production database crashed; recovered only after manual restart.

**Timeline:**

- 2026-06-15 06:08:11 UTC: Postgres crashed with ENOSPC + Read-only file system
- 2026-06-15 06:36:21 UTC: Manual restart; postmaster recovered

**Service details:**

- Service: Postgres
- Volume: postgres-volume, ~5 GB, mounted at /var/lib/postgresql/data
- Region: Southeast Asia (Singapore)

**Error logs:**

```

2026-06-15 06:08:11.419 UTC [5441] PANIC:  could not fdatasync file "00000001000000000000000D": No space left on device

2026-06-15 06:08:11.450 UTC [16] LOG:  client backend (PID 5441) was terminated by signal 6: Aborted

2026-06-15 06:08:11.450 UTC [16] DETAIL:  Failed process was running: INSERT INTO "tokens" ...

2026-06-15 06:08:11.450 UTC [16] LOG:  terminating any other active server processes

2026-06-15 06:08:11.502 UTC [11023] FATAL:  the database system is in recovery mode

2026-06-15 06:08:11.526 UTC [16] LOG:  all server processes terminated; reinitializing

2026-06-15 06:08:11.599 UTC [16] PANIC:  could not open file "global/pg_control": Read-only file system

2026-06-15 06:08:28.918 Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/54b636e6-d64e-45b4-a9dc-890a29e82a49/vol_i4rgvkpal1jhollm

2026-06-15 06:36:21.446 Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/54b636e6-d64e-45b4-a9dc-890a29e82a49/vol_i4rgvkpal1jhollm
```

**Key observations:**

- Volume was only 6% full (~307 MB used) at time of crash
- Disk usage metrics show flat usage across the incident window (no climb toward capacity)
- WAL is bounded: max_wal_size=1024MB, archive_mode=off, wal_keep_size=0, no replication slots
- The ENOSPC + Read-only remount sequence suggests a host-level storage fault, not logical quota exhaustion

**Questions for Railway:**

1. Were there any I/O errors, device full conditions, or inode exhaustion on the host node at 06:08 UTC?
2. Was the backing storage healthy before and after the incident?
3. Did fsck/journal replay complete cleanly? Any data durability concerns?
4. Are there known issues with thin-provisioning or noisy neighbors in the Singapore region?

**Mitigation taken:**

- Enabled auto-restart on crash (ON_FAILURE, max 5 retries)

~~P.S: THIS THREAD IS RELATED TO RAILWAY SERVICE BUT I AM NOT ABLE TO SELECT OTHERWISE IN THE DROPDOWN!~~