a month ago
Project: Daily_Baltimore_News_Quiz (790670f9-9194-4135-ab8a-8a6ecc5eb2ac)
Environment: production (6ebfa0a4-e8c2-48a7-a8c4-a02c90205ff7)
Service: Postgres (2ca55bdd-d89b-4f00-890b-65eb13f3f479)
Plan: Hobby
The problem I'm trying to solve
Sometime between Aug 27–28, 2026, two keys in our app's key-value table (scores and progress — player scores, daily totals, per-question answers) were wiped in the production database. Everything else (quiz content, subscribers, etc.) is intact. I've confirmed this isn't something our own application code did — nothing in our app logic performs a bulk delete on those keys.
There's a manual backup on the Postgres service, "Pre-Security-Patch Backup" (~265 MB), which should predate this by several days — its relative timestamp read "6 days ago" on Aug 29 and "7 days ago" on Aug 30, putting its actual creation around Aug 22–23.
The bug blocking me
I can't interact with that backup at all on the Backups tab:
Clicking or hovering the "Pre-Security-Patch Backup" entry does nothing — no menu, no staged-changes screen.
The cursor shows a disabled/not-allowed state when hovering over it.
The browser console throws this on interaction: Uncaught Error: Minified React error #421 (Suspense boundary hydration mismatch)
This happens consistently after a hard refresh and in a fresh/incognito session, so it's not a caching issue. I'm on Hobby, which should be sufficient for volume backup restores. The underlying volume instance's state read READY when last checked.
What I'd like
Either a fix/workaround for the Backups tab so I can complete a normal staged restore, or someone on your end restoring "Pre-Security-Patch Backup" using that same safe process — not a raw volumeInstanceBackupRestore API call, since I understand that executes immediately with no staging step and takes destructive parameters (wipeServiceIds) I don't have enough context to set correctly.
I understand this restore swaps the volume under the live Postgres service rather than standing up a sibling, so the site will serve stale data for the duration — I'm planning around that, just need the restore to be executable.
One more thing, lower priority
A separate live service, delete-corrupted-dist, runs DELETE FROM store WHERE key = 'dist' against this same database on every deploy. Last deployed ~5 months ago but still live. Unrelated to the issue above, but flagging before I delete it as cleanup.
5 Replies
a month ago
Volume backup restores are a Pro feature, so the disabled state and errors you're seeing on the Backups tab are the plan gate, not a dashboard bug. The "Pre-Security-Patch Backup" is a snapshot we took automatically before a Postgres security patch and it's retained for 30 days, but restoring it requires upgrading to Pro ($20/month). Once on Pro, the restore control becomes available and you can run it yourself. The snapshot holds the volume exactly as it was immediately before the patch, so it only helps if the missing data was still present at that point.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
Volume backup restores are a Pro feature, so the disabled state and errors you're seeing on the Backups tab are the plan gate, not a dashboard bug. The "Pre-Security-Patch Backup" is a snapshot we took automatically before a Postgres security patch and it's retained for 30 days, but restoring it requires upgrading to Pro ($20/month). Once on Pro, the restore control becomes available and you can run it yourself. The snapshot holds the volume exactly as it was immediately before the patch, so it only helps if the missing data was still present at that point.
a month ago
Thanks so much for your help. The backup is named 'Pre-Security-Patch Backup,' not the 'Auto Update {image}' pattern your docs describe for Pro-only auto-backups, so I don't think this is that feature. Can you check specifically why this backup's restore control is disabled? Thanks again.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
To correct our earlier reply, the "Pre-Security-Patch Backup" is indeed a Railway-created snapshot taken before a Postgres security patch, not the "Auto Update" backups that the docs describe as Pro-only. That distinction is real, but restoring any volume backup, including Railway-created snapshots, still requires Pro ($20/month). On Hobby the expected behavior is a Pro upgrade banner on the Backups tab, not a disabled control with a React rendering error. The React #421 error you are seeing is not the normal plan gate, so we are not able to rule out a UI issue on that specific entry. Upgrading to Pro would both clear the plan gate and, if there is a rendering issue, give us a clearer signal on whether the restore control works as expected once the gate is removed.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Thanks again. Does this mean that something went wrong with Railway that caused my project to lose its historical data? This is a real concern going forward.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
The Postgres logs from Aug 28 show only routine checkpoint activity with no errors, crashes, or restarts, and no host-level maintenance affected this service in the surrounding window. The security patch on Aug 23 restarts the container against the same volume and does not modify stored data, so the patch itself would not have removed those keys. The volume is intact and in a READY state. We don't have evidence of a Railway-side event causing this data loss. Going forward, upgrading to Pro ($20/month) unlocks native volume backups and point-in-time recovery on the service, which would let you self-restore after in-place data loss like this.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved dhconn • about 1 month ago