2 hours ago
Summary
I ran my first point-in-time recovery restore and the restored service crashes on
startup with "No space left on device", even though its volume is almost empty.
The source database is fine and still running — this is only about the restore.
Environment
Workspace: Earth (subscriptionModel: USER / Hobby)
Restored service: Postgres-restored-20260830-1402
Deployment: cef9f18b
Region: asia-southeast1-eqsg3a
Source service: Postgres (PITR enabled, archiver enabled, bucket wired)
What I did
In the source Postgres service's Backups tab I clicked "Restore to this moment"
with the latest available moment (2026-08-30 14:02 UTC). Railway created the new
service and a 5 GB volume as expected.
What happened
Recovery got most of the way through: pgBackRest pulled the base backup from S3,
restored dozens of WAL segments successfully, and reached a consistent recovery
state. It then died on a file write:
2026-08-31 04:18:47.048 P00 ERROR: [042]: unable to get 00000001000000010000001F:
repo1: 18-1/.../00000001000000010000001F-fc54998c...zst [FileWriteError]
unable to write '/var/lib/postgresql/data/pgdata/pg_wal/RECOVERYXLOG'2026-08-31 04:18:47.162 UTC [88] LOG: consistent recovery state reached at 0/4000158
2026-08-31 04:18:47.170 UTC [88] FATAL: could not extend file "base/16384/16627_fsm":
No space left on device2026-08-31 04:18:47.170 UTC [88] HINT: Check free disk space.
2026-08-31 04:18:47.176 UTC [55] LOG: startup process (PID 88) exited with exit code 1
2026-08-31 04:18:47.182 UTC [55] LOG: shutting down due to startup process failure
2026-08-31 04:18:47.210 UTC [55] LOG: database system is shut down
Why "no space" looks wrong
The volume is nearly empty. From the CLI:
{
"name": "postgres-restored",
"serviceName": "Postgres-restored-20260830-1402",
"mountPath": "/var/lib/postgresql/data",
"currentSizeMB": 207.167488,
"sizeMB": 5000,
"status": "Ready"}
Also, in the deploy logs the line
Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/315469d3-.../vol_5c5u7ei2gsww2ppx
appears AFTER the crash, not before it. That makes me suspect recovery starts
writing before the volume is mounted, so the writes land on the container's
ephemeral filesystem and fill it up.
I redeployed the service to retry. It failed the same way — this time replaying
from an earlier segment (000000010000000000000089 onwards), with the same
FileWriteError on pg_wal/RECOVERYXLOG.
Questions
-
Is PITR restore supported on the Hobby plan? The Backups tab shows a notice
saying "Backups and point-in-time recovery (PITR) are only available for
customers on the Pro plan", yet PITR archiving is enabled and running on my
account, and the restore was allowed to start. If restore is Pro-only, could
the UI block it rather than letting it fail this way?
-
If it is supported, is this a known issue with volume mount ordering during
restore, and is there a workaround?
I have left the crashed service in place so you can inspect it. Happy to provide
full logs or run anything you need.
Note: I'm on the Hobby plan so "Ask Railway Privately" is unavailable to me —
posting publicly instead. Happy to move this to a private channel if you'd
prefer, given it involves my database backups.
1 Replies
an hour ago
Volume backups and point-in-time recovery are both Pro-plan features, so the restore on a Hobby workspace is not expected to succeed. Your observation that the UI allowed the restore to start rather than blocking it is valid feedback, and the "No space left on device" crash you are seeing is consistent with the restored service writing to the container's ephemeral filesystem rather than the volume. Your source Postgres service is unaffected by any of this.
Status changed to Awaiting User Response Railway • about 1 hour ago
Status changed to Solved foundstonesp • about 1 hour ago