Database data gone
emprops-saas
PROOP

7 months ago

At Fri, May 16, 9:56 PM PST I got an error on one of my services saying that it couldn't access the database. Look at it right now, I can see that the database is completely empty and all data is GONE. No data, no backups, nothing.

My database is a postgres service with a volume attached.

Project ID: 7afe0f47-6962-48b9-8626-3e1bc3ed39f0
Environment ID: ddabfccf-fbb6-46f3-8e5c-cea5e1c99bc2

Solved

5 Replies

brody
EMPLOYEE

7 months ago

Hello,

Please provide a direct link to the database in question



emprops-saas
PROOP

7 months ago

Is it related to Railway Metal upgrade?


emprops-saas
PROOP

7 months ago

can someone help here @here


emprops-saas
PROOP

7 months ago

Hi, any updates?


vielmaoliver
HOBBY

7 months ago

Hi im facing the same issue over here. My data is gone. I was using a mongodb service


emprops-saas
PROOP

7 months ago

Hi guys I really need help with this


emprops-saas
PROOP

7 months ago

we are talking about a production app here


adam
MODERATOR

7 months ago

!t


adam
MODERATOR

7 months ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response adam 7 months ago


adam
MODERATOR

7 months ago

your issue is likely different. Please open your own thread


emprops-saas
PROOP

7 months ago

I see that a escalation was created and I see a "resolve thread" button. Am I supposed click that or something? What are my next steps?


adam
MODERATOR

7 months ago

The next step is to wait for the team to reply. There is no action you can take on your part at this time


Railway
BOT

7 months ago

Hello!

We're acknowledging your issue and attaching a ticket to this thread.

We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.

Please reply to this thread if you have any questions!


Status changed to Awaiting User Response Railway 7 months ago


jake
EMPLOYEE

7 months ago

I THINK I've found your drive. I'm going to attempt to restore it into another instance called (Postgres Restore)


jake
EMPLOYEE

7 months ago

(I'll root cause what happened after we resolve this issue)


jake
EMPLOYEE

7 months ago

@luiseel I believe that your database has been restored. I see 3 emprops_ databases*


jake
EMPLOYEE

7 months ago

Can you please validate at your earliest convenience?


jake
EMPLOYEE

7 months ago

I request that you do NOT delete the old instance so that we can perform foresics on what would have gone wrong on the old one


jake
EMPLOYEE

7 months ago

(I believe I know the root cause however, I need to be certain)


emprops-saas
PROOP

7 months ago

Sure!


jake
EMPLOYEE

7 months ago

I've DM'd you a screenshot of the Databases in that database

Thank you for confirming they look accurate. Please, check the data and also confirm


jake
EMPLOYEE

7 months ago

(Again, at your earliest convenience)


jake
EMPLOYEE

7 months ago

(Again, at your earliest convenience)


emprops-saas
PROOP

7 months ago

Ok, all data is back


jake
EMPLOYEE

7 months ago

Perfect. We'll have a look at what happened here, but it looks like it selected a snapshot of an empty database. Did you happen to have an old one that you might have deleted? If not, we'll look into it


emprops-saas
PROOP

7 months ago

Actually, I might have but I don't quite remember


okvic77
PRO

7 months ago

I have the same issue. I I'm looking at the logs and this happened when the migration to metal happened. I don't see the rollback button to restore it. It is a Postgres image.

https://railway.com/project/97f62f00-0e74-4d01-bd0f-2ee465f6b868/service/47751cac-9559-4597-9fa5-a3df17ce0ebf/data?environmentId=12cbdc22-75e1-4fc3-a83c-996de21bc5fa


Status changed to Awaiting Railway Response Railway 7 months ago


okvic77

I have the same issue. I I'm looking at the logs and this happened when the migration to metal happened. I don't see the rollback button to restore it. It is a Postgres image.https://railway.com/project/97f62f00-0e74-4d01-bd0f-2ee465f6b868/service/47751cac-9559-4597-9fa5-a3df17ce0ebf/data?environmentId=12cbdc22-75e1-4fc3-a83c-996de21bc5fa

brody
EMPLOYEE

7 months ago

Please open your own thread.


Status changed to Awaiting User Response Railway 7 months ago


jake
EMPLOYEE

7 months ago

^^This is quite odd because, this user has always been on metal


jake
EMPLOYEE

7 months ago

It looks like 7 weeks ago you deleted the volume

1374468015097381000


Railway
BOT

7 months ago

✅ The internal ticket Customers reporting data loss on volume migrations has been marked as completed.


Railway
BOT

6 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 7 months ago


Loading...