11 days ago
Railway just automatically updated my postgres deployment to apply a CVE patch. After that, my DB was unresponsive. There was also an incident (https://status.railway.com/incident/RL8FRJE6) around the same time. Now the incident seems to be resolved, but the postgres deployment in question won't redeploy. It gets stuck at "creating containers" every time.
I have another postgres deployment in my project, and I just tried redeploying it, and it worked just fine. I'm worried that my DB volume got corrupted or something in the process of the automatic update. I'm seeing a log in the deploy that happened right after the automatic update that says, "database system was not properly shut down; automatic recovery in progress"
I have automatic backups turned on, but all the backups listed in that tab say that they're <1 MB. My volume has 3 GB of data. Are my backups healthy?
I don't know what to do here and I just want to restore my database and my project to the state it was in before the automatic update and/or incident. Please help!
9 Replies
11 days ago
Apologies for the trouble. We had a connectivity incident affecting US West that caused deployments and databases in that region to become slow or unreachable, which is the likely cause of your Postgres deployment getting stuck at "creating containers." The region is currently recovering, and you can follow updates at the incident page. Your volume still holds ~3 GB of data, so your data is not lost. Regarding your backups showing <1 MB, that is expected - backups are incremental and copy-on-write, so the listed size reflects only the data that changed since the previous snapshot, not the total recoverable data. A restore from any of those backups would include the full volume contents. The "automatic recovery in progress" log is normal Postgres behavior after an unclean shutdown and does not indicate corruption. Additionally, a pre-patch volume snapshot was taken automatically before the security update, which you can find in the service's Backups tab.
Status changed to Awaiting User Response Railway • 11 days ago
11 days ago
I can't seem to restore any of my backups. Nothing happens when I click on them in the backups tab and the button to restore isn't there anymore. I think I'm going to need help from Railway staff to get things back to the way they were before.
Status changed to Awaiting Railway Response Railway • 11 days ago
11 days ago
The backup restore issue is connected to the same US West incident, which is still recovering. Your Postgres volume is currently in an error state from the outage, which is preventing the restore UI from functioning. Your data (~3 GB) is still on the volume and is not lost. Once the volume state recovers, backup restore should become available again
Status changed to Awaiting User Response Railway • 11 days ago
11 days ago
Ok, my postgres deployment has recovered and everything seems to be back to normal. The incident has resolved, yet my volume is still in a "migration failed" error state. My backups UI is still unusable. Any idea how to resolve the "migration failed" error state?
Attachments
Status changed to Awaiting Railway Response Railway • 11 days ago
11 days ago
Could you link the db that you're hitting this on? Would love to make sure i have the right one
Status changed to Awaiting User Response Railway • 11 days ago
11 days ago
I believe it's the one that I attached to this ticket when I created it. Service ID is 6dec1fcf-fdc5-49bc-beaf-f291d4e7d7d5
Status changed to Awaiting Railway Response Railway • 11 days ago
11 days ago
Didnt originally come through but see it now, just poked it on our end and looks healthy here. Could you take a peek and let me know how it looks on your end?
Status changed to Awaiting User Response Railway • 11 days ago
11 days ago
We're so back. Thanks!
Status changed to Awaiting Railway Response Railway • 11 days ago
Status changed to Solved trevorblades • 11 days ago
Status changed to Awaiting User Response Railway • 11 days ago
4 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 4 days ago