🚨 URGENT DATA LOSS: Railway forced migration to Metal wiped my production MySQL database
davebecherer2003
PROOP

a year ago

Project- CNTG-VIDEORECOMMENDER-BASE

Issue: upgrade to Railway Metal completely reinitialized my MYSQL database. My tables are gone, which represent about 2 years of work.

Project ID: 421de913-10a8-48a4-9c91-a736510d543d

Environment ID: 39809c9a-bceb-4019-a943-dcc01d4e9ba0

Is it possible that there is a backup on your end and a recovery team that can please assist?

<@&743878049736687688> Team - im reading the guide ansee that it is possible that hte migration is still underway?

30 Replies

davebecherer2003
PROOP

a year ago

@Percy - Thank you for acknowledging this!

Can you confirm:

  1. Is the Metal migration still in progress for my database?
  2. Is my production data being transferred in the background?

This is affecting my live application with paying customers.


a year ago

You obviously had this data on a volume, right?


a year ago

Redeployment was never an issue?


a year ago

To be clear, migration is not still underway. It has completed for your database.


davebecherer2003
PROOP

a year ago

@Loudbook

i have a volume, that looks empty. it may have never been attached. This may be my problem. I set this up 2 years ago when I knew very little about it. I've been iterating and building on it since then. I do have some other places I can rebuild those tables from and this may be a terrible lesson learned for me that backups are not happening without attached volumes. I thought I had taken care of it with that holiday-volume in my project but it doesn't appear to be mounted and it seems empty.


a year ago

If it’s unmounted and it never was, then your data is gone.


a year ago

No backups are taken by Railway. Your database is your responsibility.


davebecherer2003
PROOP

a year ago

i dont know if this is working

1383927816218742924


davebecherer2003
PROOP

a year ago

@Loudbook - I understand the volume issue for the future. Right now I'm focused on recovery options, but I'm confused about what happened based on your documentation.

Your Metal migration docs state:

"For Stateless deployments, meaning, a deployment with no volume- there is no downtime. Stateless deployments are just landing into a new region."

But what I experienced was complete database reinitialization and total data loss - which doesn't match what should happen for a stateless migration.

This discrepancy makes me wonder if there might be recovery options I'm not aware of:

  1. Since this didn't follow the normal stateless migration process, are there any automatic snapshots or backups Railway takes during failed migrations?
  2. Could there be any remnants in the old region that weren't properly migrated?
  3. Are there any system logs or temporary files from the migration process that might contain data?
  4. Does Railway keep any rollback data when migrations don't go as documented?

I'm not looking to assign blame - I just need to know if the unexpected behavior means there might be ANY possibility of data recovery before I start rebuilding from scratch.

Even partial data or table structures would be helpful at this point. in the future of course i'll make sure the volume is attached, and as much as I am to blame here for not undersatnd how this works, I'm hoping that there is SOME remedy as I've got several sites relying on these tables that are now down.


davebecherer2003
PROOP

a year ago

@Loudbook - One more question about rollback:

Since your docs say stateless migrations are "just landing into a new region," if I rollback to the old region, would I reconnect to my original container/data?

Or does Railway immediately destroy the old deployment when migrating?

Given that my migration didn't behave like the documented stateless process, I'm wondering if my original data might still exist in the old region.


a year ago

There is no backup. Your data is gone. Period.


a year ago

The fact you managed to run it ephemeral for 2 years is crazy btw


davebecherer2003
PROOP

a year ago

the documentation specifically mentions rollbacks for failed migrations. and thanks for all your encouraging words, your customer service skills are top notch.


a year ago

Your migration did not fail.


davebecherer2003
PROOP

a year ago

are you actually on staff for are you just "helping out"


a year ago

Oh no I'm not staff


davebecherer2003
PROOP

a year ago

exactly.


a year ago

But that doesn't change the truth of my words lol


davebecherer2003
PROOP

a year ago

yoiu're just a troll.


a year ago

And staff are actively monitoring, they would correct me if they saw something I said was incorrect


davebecherer2003
PROOP

a year ago

@Percy this should have come under the umbrella of "stateless migration" I'm thinking. It is possilbe that the data still exists at the old endpoint (legacy server). if i can get to it and mount a volume I have a chance to correct my issue.


a year ago

Percy is a robot


a year ago

Percy does not communicate


a year ago

And this is not a stateless migration


a year ago

Because your service was not stateless


a year ago

The old deployment was destroyed as soon as the new one was created


davebecherer2003
PROOP

a year ago

not true their documentation tells you that you can roll back.


a year ago

rollbacks for failed migrations.


a year ago

Railway knows if your migration failed or not


a year ago

And the fact that your data wasn't actually saved anywhere removes any chance of even a failed deployment rollback


Welcome!

Sign in to your Railway account to join the conversation.

Loading...