a month ago
My staging PostgreSQL volume reached its 500 MB limit during a disposable derived-data projection backfill. PostgreSQL now cannot complete WAL recovery because it cannot allocate pg_wal/xlogtemp.
Project: dramaos-staging
Project ID: fbd174f5-1e83-4d7b-b050-7d7811e20f9f
Environment: staging
Environment ID: 4dd916ff-93ce-482f-9cb2-c82463875b73
Service: Postgres
Volume: postgres-volume
Volume instance ID: 6ecc4894-ef0d-4be6-b217-b63cebf8a8e6
Before the incident, the canonical database occupied approximately 160 MB. The volume is now approximately 492.19 MB. The additional data is an explicitly disposable derived table:
YouTubeReportingMetricProjection
PostgreSQL logged:
- could not extend file "base/16384/22486": No space left on device
- The failing statement was an insert into YouTubeReportingMetricProjection
- could not write to file "pg_wal/xlogtemp...": No space left on device
Please provide one of the following without permanently upgrading the plan:
- Temporary recovery headroom sufficient for PostgreSQL to complete WAL recovery and allow us to truncate the disposable projection;
- A safe export/snapshot of the existing stopped staging volume for offline recovery; or
- A support-assisted recovery method that preserves the existing database.
We will:
- create and validate a logical dump before replacing anything;
- truncate only YouTubeReportingMetricProjection;
- preserve all canonical application, OAuth, provider, ingestion, provenance, and historical records;
- retain all immutable raw Reporting artifacts already stored externally in R2;
- avoid pg_resetwal, valid-WAL deletion, or direct relation-file modification.
There are currently no Railway volume backups or PITR recovery points for this staging volume.
Production must not be accessed, modified, restored, or redeployed.
Please do not resize or replace the volume without confirming the proposed recovery procedure first.
3 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
The volume itself is healthy and still holds your data, so this is a full-disk stall, not data loss. Postgres can't finish recovery only because the volume is at its 0.5 GB limit with no free space left to write its write-ahead log.
On the Trial plan volumes are fixed at 0.5 GB, so they can't be resized or backed up, and we don't provide temporary increases to get a database booting again. Adding the headroom Postgres needs to finish recovery means moving to a paid plan, where the volume can be resized to give it room.
Status changed to Awaiting User Response Railway • about 1 month ago
dizzydes90
The volume itself is healthy and still holds your data, so this is a full-disk stall, not data loss. Postgres can't finish recovery only because the volume is at its 0.5 GB limit with no free space left to write its write-ahead log. On the Trial plan volumes are fixed at 0.5 GB, so they can't be resized or backed up, and we don't provide temporary increases to get a database booting again. Adding the headroom Postgres needs to finish recovery means moving to a paid plan, where the volume can be resized to give it room.
a month ago
Thanks for confirming that the volume itself is healthy and the data is still intact.
Before considering a paid resize, could you please clarify one remaining recovery option?
Is there any supported way on the Trial plan to export or download the stopped PostgreSQL volume as-is for offline recovery, without first booting PostgreSQL or resizing the volume?
We only need to recover it on a larger local filesystem, truncate the disposable derived table, create a logical dump, and then restore the compact database into a fresh 0.5 GB staging volume.
We do not need temporary Railway storage headroom if a raw volume export/download is possible.
If this is not supported on Trial either, please confirm that explicitly.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
There is no supported way to export or download a volume's raw contents offline. Volumes are not downloadable as disk images on any plan. The only path to recover this data is upgrading to a paid plan (Hobby at $5/mo), resizing the volume to give Postgres the headroom it needs to complete WAL recovery, then running pg_dump yourself once it boots.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago