Hey there! We've found the following might help you get unblocked faster:

• 🧵 Postgres Database was wiped?

• 🧵 Issues after renaming the database

• 🧵 Postgres database server is crashing after adding timescaledb extension

If you find the answer from one of these, please let us know by solving the thread!

Hello,

I understand this is an urgent situation. Unfortunately, we're unable to restore data when the issue stems from deployment configuration or user actions rather than a platform failure.

For situations like this, we always recommend maintaining regular backups of critical data to ensure quick recovery.

Best,

Brody

Hello Brody,

Thanks for the quick response. We tried to replicate the issue and the deployment was successful (data didn't get wiped), this seems like the issue comes from Railway and not from the deployment itself, can you assist?

Hey Brody,
Presumably Railway as a company holds routine snapshots of the databases in question here? - I understand a company policy on not restoring the data when it stems from deployments. But regardless of company policy I can only assume you still hold backups your end?

Hi Brody

Anything you can do to help at this point would be greatly appreciated, as we do not believe this is a deployment issue.

We only host disaster recovery snapshots for our use, there is unfortunately nothing we can do here since you don't have database backups configured on that service.

Hi Angelo

What's the criteria for you to assist us and help us get our data back?

We've identified that the data lose was not caused by our end, meaning we can only assume it's a malfunction from Railway. We have postres data from over 600 users which has literally dissapeared without any log, as if it never existed. Please can you do some form of investigation to check why this has happened?

We tested all scenarious on development and our data did not get wiped. We've tested again on both staging and production and our data has not been wiped again, yet, our data has gone missing. Please do something to assist us.

We deployed a new server and the database got wiped

I am terribly sorry, but I don't see how there is a Railway feature here that would have caused your database to be lost.

If there was a configuration that connected to your production database, then the boundary of responsibility is on your end. If the service was randomly deleted or something that was clearly a Railway bug, then we would have cracked open a DR snapshot. However, to my knowledge, no such action occurred, nor did you have native backups enabled. I am terribly sorry for the business in-continuity you face, but there is nothing further that we can do.

This is disappointing because your explanation sounds like you could open the DR snapshot and offer us assistance?

These things do happen and I do understand that not having a back-up through railway was a clear mistake, but if you do have a DR snapshot of our database and you are able to restore any of our data the difference this would make to our business at this stage would be immense.

As well, any insight you can offer into what went wrong will avoid this happening in the future. There are zero logs on the Railway portal explaining why our postres database would wipe, especially because the roll we created had nothing to do with postres, as tested on development.

The DR snapshot reverts everyone on that host to that point and time. It's not intended for pulling the data out for a single customer, unfortunately.

Was there a staged migration that you had?

Hi Angelo,
If you do have a DR snapshot, presumably it would be possible to stand that snapshot up in an isolated environment and pull our data off it? As Alex has mentioned this is really important data for us so anything you can do would be greatly appreciated. It sounds like its possible but not common practise for Railway if you're not at fault? Which if fair enough but what i'm reading is its still possible - In theory the DR snapshot stands up to before our deployment, that data would still be on it, which could be extracted and reinstated for us.

There were some changes but not a staged migration no.

Angelo, we would be willing to pay to restore our database.

Hello,

We do have snapshots, yes, but as Angelo mentioned, they are not used to restore user-initiated deletions.

It's policy mainly because it is not a simple, straightforward, or fast process to pull out an individual volume's data from the snapshot, as they are snapshots from the parent disk, not snapshots of each individual volume.

So with that said, my apologies, but we will not be accommodating this request.

Best,
Brody