24 days ago
Production Postgres is down in a WAL recovery crash loop after the 5 GB Hobby volume filled during a large JSONB metadata backfill.
Project: attractive-elegance
Project ID: 7479d1e1-719d-42a3-b131-2fac737d51d1
Environment: production
Environment ID: 06ec966d-48de-466c-b3c9-f3f718cd21f4
Postgres service ID: 5e14ff2e-af3d-4911-a811-316b74dabdff
Volume ID: 0467ad36-c439-43c7-9a8c-57aa0cceef09
Volume: postgres-volume, 5000 MB Hobby maximum
Affected API service: edgetracker-api
Repository: https://github.com/aidanreidgilbert-ux/Misprice
Repeated Postgres failure after redo completes:
FATAL: could not write to file "pg_wal/xlogtemp.47": No space left on device
The original write also reported SQLSTATE 53100 while creating pg_wal/xlogtemp.*.
The Railway CLI reports 4162.01 MB current data, while the Postgres boot log reports "pgbackrest: volume 4614 MiB" and the filesystem has no recovery headroom. Postgres cannot reach a consistent state, so the API deployment cannot start. There are no Railway volume backups or backup schedules configured.
Please preserve the existing volume and data. Do not reset/delete WAL or restore an empty database. Could Railway temporarily expand/repair the filesystem or provide emergency recovery headroom without requiring a permanent Pro upgrade? Similar Hobby incidents have been recovered by Railway staff with a one-time filesystem/volume intervention.
We have already merged application safeguards: bounded 100-row statements, a 250-row/provider backfill cap, transient recovery retries, and a fail-closed 75% volume-capacity guard. Once Postgres starts, we will immediately remove only regenerable derived data using SQL, configure retention, and keep usage below the 5 GB Hobby limit.
1 Replies
Status changed to Awaiting Railway Response Railway • 24 days ago
24 days ago
Resolved without a plan upgrade. We paused the service, verified local byte-for-byte backups of pg_control and all required WAL segments, then ran normal PostgreSQL 17 crash recovery with wal_init_zero=off so the next WAL file could be created sparsely. Recovery completed cleanly; no WAL segment was deleted or reset. We exported the regenerable PriceHistory table (1,496,586 rows) to a validated custom-format archive, truncated only that derived telemetry cache, and preserved users, analyses, positions, trades, and other product data. Postgres is now healthy with 1.6 GB free (66% used), the API readiness check is green, and application guards are deployed to keep one hot day of history and stop discretionary history/metadata writes at the capacity threshold. No Railway staff action is needed now. Thank you.
Status changed to Solved aidanreidgilbert-ux • 24 days ago