a day ago
Hi Railway team,
We need help validating a PostgreSQL PITR restore on the Hobby plan.
Our application has not been deployed. Database migrations are on hold until we can complete and verify a restore.
- Restore fills the volume
On October 22, we attempted a PITR restore into a separate temporary service with a 5 GB volume.
Logs showed that recovery reached the target transaction, but startup then failed with:
FATAL: could not write to file "pg_wal/xlogtemp.70": No space left on device
The service restarted repeatedly and never accepted connections. We deleted the temporary service.
Logs showed hundreds of WAL segments being fetched, although the target appeared to require only a few. Our hypothesis is that WAL fetching or retention beyond the recovery target filled the volume. We could not inspect the filesystem to confirm this.
- On-demand backups fail
Two attempts to create an on-demand backup using the Railway CLI returned:
{"code":"GRAPHQL_ERROR","error":"Failed to create a backup","hint":null}
No backup started in the service logs. The CLI backup list was empty despite automatic backups existing.
- Current observations
The automatic differential backup on October 27 completed successfully at approximately 06:24 UTC, followed by a PITR anchor.
Recent archive-push operations also completed successfully. However, we still have no restore verified by connecting to the restored database and comparing its data.
An earlier archive authentication failure caused a WAL gap on October 21–22. A new full backup on October 22 began a new chain.
The cause of that authentication failure remains unconfirmed. Archiving resumed after a database redeploy.
- Questions
- What is the supported procedure to validate a PITR restore within the Hobby 5 GB volume limit?
- Can WAL fetching or retention during recovery be bounded to what is needed for the recovery target?
- Why does on-demand backup creation fail, and how can we diagnose it?
- Why does the CLI backup list show no backups while automatic backups exist?
- Are there known PITR or SSH limitations related to the ongoing incident?
We can provide sanitized log excerpts and service details privately if needed.
Please advise before we attempt another restore.
Thank you.
2 Replies
Status changed to Awaiting Railway Response Railway • about 24 hours ago
a day ago
The restore crash is the restored volume running out of room during write-ahead log replay. A PITR restore lays down the base backup and replays every WAL segment between it and your target, and the volume must hold the data plus roughly all of that WAL at once, so the space needed depends on how much WAL sits between backup and target, not on database size. That explains the hundreds of segments, and it means replay is bounded by choosing a target close to a recent backup: a target shortly after the October 27 differential (around 06:24 UTC) needs far less room than one reaching back toward the October 22 full. A failed restore never touches your source database.
Your volume is already at the 5 GB maximum Hobby allows, so the restored volume cannot be grown on this plan; a restore needing more room requires a plan with a higher volume ceiling. PITR restore itself is supported on Hobby.
The automatic full and differential backups are PITR backups, a separate system from volume backups, so they do not appear in the volume backup list the CLI shows. The cause of the "Failed to create a backup" response on the on-demand attempts has not been established.
Status changed to Awaiting User Response Railway • about 24 hours ago
Railway
The restore crash is the restored volume running out of room during write-ahead log replay. A PITR restore lays down the base backup and replays every WAL segment between it and your target, and the volume must hold the data plus roughly all of that WAL at once, so the space needed depends on how much WAL sits between backup and target, not on database size. That explains the hundreds of segments, and it means replay is bounded by choosing a target close to a recent backup: a target shortly after the October 27 differential (around 06:24 UTC) needs far less room than one reaching back toward the October 22 full. A failed restore never touches your source database. Your volume is already at the 5 GB maximum Hobby allows, so the restored volume cannot be grown on this plan; a restore needing more room requires a plan with a higher volume ceiling. PITR restore itself is supported on Hobby. The automatic full and differential backups are PITR backups, a separate system from volume backups, so they do not appear in the volume backup list the CLI shows. The cause of the "Failed to create a backup" response on the on-demand attempts has not been established.
an hour ago
Thank you. Before retrying, could you clarify how to select the differential backup as the restore starting point?
In our failed attempt, the requested target was later than the backup, but because the source was idle, Railway changed it to recovery_target_xid=3023, the anchor immediately after the full backup. Logs showed that target being reached, yet hundreds of WAL segments were fetched and the volume filled before startup.
If we choose a target immediately after a recent differential:
- Will Railway use that differential plus its parent full, rather than replaying WAL from the full backup?
- Can WAL fetching continue beyond the selected target? If so, how can we prevent the same disk exhaustion?
- What is the supported way to request a fresh PITR backup? Our on-demand command failed twice with GRAPHQL_ERROR.
We want to perform one supervised restore into a separate service within the Hobby 5 GB limit, without changing the source database.
Status changed to Awaiting Railway Response Railway • about 1 hour ago