21 hours ago
Hi Railway team,
We completed an isolated PostgreSQL PITR restore test on the Hobby plan and identified a specific disk exhaustion issue.
Our PostgreSQL database is only 41.4 MB, with a 5 GB volume.
The restore successfully reached the requested recovery target:
recovery stopping after commit of transaction 809, time 2026-11-11 05:38:16.433252+00
It then selected timeline 2 but failed during promotion:
FATAL: could not write to file "pg_wal/xlogtemp.95": No space left on device
The restore container subsequently restarted and repeated the failure.
A particularly relevant log entry was:
volume 4614 MiB; queue-max=2307 MiB
The highest observed disk usage was 4.963328 GB.
Our investigation suggests that asynchronous WAL prefetching may be consuming too much disk headroom during recovery, although we have not measured the exact allocation.
Could your engineering team clarify:
- Can the managed PITR restore image limit archive-get-queue-max (for example, to 128 MiB) or disable asynchronous WAL prefetch before the restored sibling's first startup?
- Is there an officially supported way to reserve sufficient disk space for PostgreSQL WAL processing and timeline promotion on a 5 GB Hobby volume?
- Could the restore image's queue-sizing logic be contributing to this failure?
- If these settings are not configurable by customers, is there another supported Hobby-compatible recovery method?
We would prefer to solve this without upgrading the plan, since the database itself is only 41.4 MB.
The temporary restore service and volume were safely deleted. The original PostgreSQL source and production application remain unchanged.
We can provide sanitized deployment logs if needed.
Thank you for your help.
5 Replies
21 hours ago
How much disk a restore needs depends on how much write-ahead log sits between the base backup and the recovery target you pick, not on how big the database is. The restored service has to hold the base data plus the write-ahead log it replays, so a 41 MB database that writes a lot can still need gigabytes of replay room. A failed restore never touches the source service or its volume.
There are two supported ways to give a restore more room. The first works on Hobby: choose a recovery target closer to a recent backup, so there is less write-ahead log to replay. The second is a bigger volume. When the restored service's volume is below your plan's maximum, you can grow it and redeploy the restored service, and recovery picks up where it stopped. Your workspace's current maximum volume size is 5 GB, so a restore that needs more than that would need a plan with a higher volume limit.
Status changed to Awaiting User Response Railway • about 21 hours ago
Railway
How much disk a restore needs depends on how much write-ahead log sits between the base backup and the recovery target you pick, not on how big the database is. The restored service has to hold the base data plus the write-ahead log it replays, so a 41 MB database that writes a lot can still need gigabytes of replay room. A failed restore never touches the source service or its volume. There are two supported ways to give a restore more room. The first works on Hobby: choose a recovery target closer to a recent backup, so there is less write-ahead log to replay. The second is a bigger volume. When the restored service's volume is below your plan's maximum, you can grow it and redeploy the restored service, and recovery picks up where it stopped. Your workspace's current maximum volume size is 5 GB, so a restore that needs more than that would need a plan with a higher volume limit.
21 hours ago
Thank you for the explanation. I still need help from a human Railway engineer, as our investigation identified a more specific issue.
Our restore successfully reached the exact requested transaction (XID 809) and selected timeline 2, but then failed during promotion:
FATAL: could not write to file "pg_wal/xlogtemp.95": No space left on device
The restore logs also show:
volume 4614 MiB; queue-max=2307 MiB
This suggests that asynchronous WAL prefetching or WAL retention may be consuming disk space needed for promotion. We have not verified the exact filesystem allocation.
Our database is only 41.4 MB, and we would like to retain the Hobby plan.
Could an engineer please confirm:
- Can Railway's managed PITR restore image limit archive-get-queue-max or disable asynchronous prefetch before the restored service starts?
- Is the queue size of 2,307 MiB on a 4,614 MiB effective volume expected behavior?
- Is there a supported way to reserve disk headroom for WAL processing and PostgreSQL promotion on Hobby?
- If not, could Railway suggest another supported restore procedure that works within the 5 GB limit?
We understand that restoring closer to a recent backup can reduce WAL replay, but our failed restore already reached its exact recovery target before disk exhaustion during promotion.
We are not requesting changes to our production PostgreSQL service.
Could you please escalate this to a human engineer for technical review?
Thank you.
Status changed to Awaiting Railway Response Railway • about 21 hours ago
21 hours ago
You're right that this restore reached its recovery target before it failed, so a closer target would not have changed this particular run. We can't currently confirm whether the 2,307 MiB queue size is expected for a 4,614 MiB volume, or whether the queue sizing contributed to the disk filling at promotion.
The restore runs on the image and settings that the restore process sets up. We haven't verified a customer-facing setting that changes the async WAL fetch queue or reserves disk headroom before the restored service's first startup, so we won't point you to one.
What we can confirm: the restored service has to hold the base data plus the write-ahead log it pulls in. That can use far more disk than the database itself, and on Hobby the whole budget is the volume, capped at 5 GB. A failed restore doesn't affect your source service or its volume.
Within that 5 GB cap, the only supported way to need less disk is to restore to a point closer to a recent base backup, so less write-ahead log has to be fetched and replayed.
Status changed to Awaiting User Response Railway • about 21 hours ago
Railway
You're right that this restore reached its recovery target before it failed, so a closer target would not have changed this particular run. We can't currently confirm whether the 2,307 MiB queue size is expected for a 4,614 MiB volume, or whether the queue sizing contributed to the disk filling at promotion. The restore runs on the image and settings that the restore process sets up. We haven't verified a customer-facing setting that changes the async WAL fetch queue or reserves disk headroom before the restored service's first startup, so we won't point you to one. What we can confirm: the restored service has to hold the base data plus the write-ahead log it pulls in. That can use far more disk than the database itself, and on Hobby the whole budget is the volume, capped at 5 GB. A failed restore doesn't affect your source service or its volume. Within that 5 GB cap, the only supported way to need less disk is to restore to a point closer to a recent base backup, so less write-ahead log has to be fetched and replayed.
3 hours ago
Could a Railway engineer review this as a managed PITR restore-image issue? The automated answer about selecting a target closer to a backup does not explain our failed run.
The selected differential backup stopped at 2026-11-11 05:38:12 UTC. Our requested target was 05:38:16.433252 UTC. PostgreSQL logged “recovery stopping after commit of transaction 809” at that exact time, then “selected new timeline ID: 2”, followed immediately by “FATAL: could not write to file pg_wal/xlogtemp.95: No space left on device”. Subsequent container starts repeated the same sequence. The 5 GB Hobby volume reached approximately 4.963 GB observed usage. The restore image logged a 2,307 MiB archive-get queue on a 4,614 MiB effective volume.
Could engineering please confirm:
- What consumed the volume before timeline promotion, particularly the archive-get queue versus pg_wal? Is the 2,307 MiB queue size expected for this restore image?
- Is there a supported way to limit archive-get prefetch or reserve promotion headroom before the restored sibling’s first start, without changing the source database?
- If the managed restore cannot do that, what supported Hobby-compatible recovery procedure would Railway recommend for this exact failure?
Please escalate beyond the automated response. We can provide sanitized deployment logs and identifiers privately. The failed sibling and volume have been deleted; the source database and production application are unchanged. We will not repeat the restore unchanged.
Status changed to Awaiting Railway Response Railway • about 3 hours ago
an hour ago
Your run reached its recovery target and then ran out of disk during promotion, so restoring closer to a backup doesn't explain this failure.
The queue-max figure is computed by the Postgres restore image. Since the restored volume has been deleted, we can't break down how its space was split between the archive-get queue and pg_wal, or say whether a 2,307 MiB queue on a 4,614 MiB volume is the intended sizing for that image.
We haven't verified any customer-facing setting that limits that prefetch or reserves promotion headroom before the restored service's first start. The restored service is configured from the source service's configuration.
On Hobby, the restored service's volume is the entire disk budget, and your workspace is already at its 5 GB maximum, so the volume can't be grown in place to let a stuck restore continue. Within that limit, we don't have another supported restore procedure to point you to for this failure. A failed restore never affects your source database or its volume.
Status changed to Awaiting User Response Railway • about 1 hour ago