Lost data from east region to west region. Wondering if there's any way to rollback
zackafter
PROOP

a year ago

With the big issue that's happened over the last hour, I tried to move regions from East to West. While this worked for all the other services, our mongoDB env was completely dropped. Is there any way to recover that data?

Looking specifically for staff from the Railway team to see if there's anything on your side to help us with this.

Solved

50 Replies

zackafter
PROOP

a year ago

5fd7c146-2bc8-4c17-ad18-4d0b59c44fa0


brody
EMPLOYEE

a year ago

!t


brody
EMPLOYEE

a year ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response brody over 1 year ago


zackafter
PROOP

a year ago

Hoping that y'all have access to some other MongoDB interface from the other region.

My guess is that you have migration scripts when moving from one region to another, but since the whole thing was down, it failed so it gave me a blank database


brody
EMPLOYEE

a year ago

is this effecting a production application with a userbase?


zackafter
PROOP

a year ago

Yes


zackafter
PROOP

a year ago

I had to migrate data from our staging environment to our production environment, as the production database was completely wiped


zackafter
PROOP

a year ago

We're in an OK spot now, but we lost a lot of data


brody
EMPLOYEE

a year ago

@Ray - production database wiped during region migration


zackafter
PROOP

a year ago

For context, we host websites for Funeral Homes. So people's obituary pages were down for a good hour

Thankfully our obits are stored in a separate datastore so none of that actual data was lost



zackafter
PROOP

a year ago

Yes that's the service


zackafter
PROOP

a year ago

I'll defer to you guys if there is a backup… or if our data is still technically available in the east region. Open to any solution as long as we can get that data back


To confirm, was this in the production environment and a migration from US East -> US West?


Is the volume name the same before and after you performed the migration?


zackafter
PROOP

a year ago

Correct, this was in the production environment. I migrated from US east to US west (as a possible solution to US east being down)

I did not change the volume name at all (I don't think I can since there is no volume attached)


brody
EMPLOYEE

a year ago

there is no volume attached…?


zackafter
PROOP

a year ago

Replicas
The number of instances that will be run for each deployment of this service.
Replicas
1
Replication is only available for services that do not have a volume attached.


zackafter
PROOP

a year ago

Unless I am misunderstanding the question… it doesn't look like we configured a volume


brody
EMPLOYEE

a year ago

screenshot of the database service please


zackafter
PROOP

a year ago

is the MongoDB Settings page sufficient?


brody
EMPLOYEE

a year ago

no


Thanks for the info - going to investigate this now.

(No worries on the volume name, I only asked to make sure I'm looking at the correct thing)


zackafter
PROOP

a year ago

thank you both


zackafter
PROOP

a year ago

@Brody do you still want that screenshot?


zackafter
PROOP

a year ago

1265068513631146000


brody
EMPLOYEE

a year ago

there is indeed a volume on the mongo service


zackafter
PROOP

a year ago

yeah that makes sense. i just don't see anywhere to name it


zackafter
PROOP

a year ago

I know these things take time but just getting a sense for timing—is this something I should expect a response about tonight or should I expect something more realistically tomorrow


zackafter
PROOP

a year ago

(I also need to know so I can have our team re-create any data that was lost)


@zackword sorry, was attempting to see if I can restore your data

Do you have anything in the database linked above currently?


Can you also give me some table names so I can confirm?


zackafter
PROOP

a year ago

We have our staging data in the DB now, I migrated it just to restore customers


zackafter
PROOP

a year ago

But I would prefer the last production from us-east if you do have that


zackafter
PROOP

a year ago

Some table names- lol I think we everything is under the “test” database, and the tables would look like users, venues, pages


Gotcha. Is it alright if I created a new database in your project so I can restore your previous data from US East into it?


zackafter
PROOP

a year ago

Absolutely


You should see a new Mongo service named "MongoDB-Restored"

https://railway.app/project/5fd7c146-2bc8-4c17-ad18-4d0b59c44fa0/service/439f896d-fefd-4946-830d-0f258383cf90

Can you check if the data is accurate?


zackafter
PROOP

a year ago

I just checked the users table and that looks right


zackafter
PROOP

a year ago

Thank you so much


zackafter
PROOP

a year ago

Guessing the best way to switch is to redeploy our app with the new mongodb service


zackafter
PROOP

a year ago

Is that what you’d recommend?


Yep, redeploy with MONGO_URI set to the new one.


zackafter
PROOP

a year ago

Got it. Thank you so much


zackafter
PROOP

a year ago

We’re going to flip the switch after doing another quick test


Super sorry about this, it should never happen. I've sent some credits your way (average of your past 3 month's bill) that will be used to offset your next bill


Please let me know if you see anything off 🙏


zackafter
PROOP

a year ago

Thank you for the credits


zackafter
PROOP

a year ago

I need to run but this is super helpful


zackafter
PROOP

a year ago

Nearly had a heart attack


Me too 😅


Status changed to Solved ray-chen over 1 year ago


Loading...