🚨 URGENT DATA LOSS: Railway forced migration to Metal wiped my production MySQL database
davebecherer2003
PROOP

9 months ago

Project- CNTG-VIDEORECOMMENDER-BASE
Issue: upgrade to Railway Metal completely reinitialized my MYSQL database. My tables are gone, which represent about 2 years of work.

Project ID: 421de913-10a8-48a4-9c91-a736510d543d
Environment ID: 39809c9a-bceb-4019-a943-dcc01d4e9ba0

Is it possible that there is a backup on your end and a recovery team that can please assist?

<@&743878049736687688> Team - im reading the guide ansee that it is possible that hte migration is still underway?

30 Replies

davebecherer2003
PROOP

9 months ago

@Percy - Thank you for acknowledging this!

Can you confirm:

  1. Is the Metal migration still in progress for my database?

  2. Is my production data being transferred in the background?

This is affecting my live application with paying customers.


9 months ago

You obviously had this data on a volume, right?


9 months ago

Redeployment was never an issue?


9 months ago

To be clear, migration is not still underway. It has completed for your database.


davebecherer2003
PROOP

9 months ago

@Loudbook
i have a volume, that looks empty. it may have never been attached. This may be my problem. I set this up 2 years ago when I knew very little about it. I've been iterating and building on it since then. I do have some other places I can rebuild those tables from and this may be a terrible lesson learned for me that backups are not happening without attached volumes. I thought I had taken care of it with that holiday-volume in my project but it doesn't appear to be mounted and it seems empty.


9 months ago

If it’s unmounted and it never was, then your data is gone.


9 months ago

No backups are taken by Railway. Your database is your responsibility.


davebecherer2003
PROOP

9 months ago

i dont know if this is working

1383927816218743000


davebecherer2003
PROOP

9 months ago

@Loudbook - I understand the volume issue for the future. Right now I'm focused on recovery options, but I'm confused about what happened based on your documentation.

Your Metal migration docs state:
"For Stateless deployments, meaning, a deployment with no volume- there is no downtime. Stateless deployments are just landing into a new region."

But what I experienced was complete database reinitialization and total data loss - which doesn't match what should happen for a stateless migration.

This discrepancy makes me wonder if there might be recovery options I'm not aware of:

  1. Since this didn't follow the normal stateless migration process, are there any automatic snapshots or backups Railway takes during failed migrations?

  2. Could there be any remnants in the old region that weren't properly migrated?

  3. Are there any system logs or temporary files from the migration process that might contain data?

  4. Does Railway keep any rollback data when migrations don't go as documented?

I'm not looking to assign blame - I just need to know if the unexpected behavior means there might be ANY possibility of data recovery before I start rebuilding from scratch.

Even partial data or table structures would be helpful at this point. in the future of course i'll make sure the volume is attached, and as much as I am to blame here for not undersatnd how this works, I'm hoping that there is SOME remedy as I've got several sites relying on these tables that are now down.


davebecherer2003
PROOP

9 months ago

@Loudbook - One more question about rollback:

Since your docs say stateless migrations are "just landing into a new region," if I rollback to the old region, would I reconnect to my original container/data?

Or does Railway immediately destroy the old deployment when migrating?

Given that my migration didn't behave like the documented stateless process, I'm wondering if my original data might still exist in the old region.


9 months ago

There is no backup. Your data is gone. Period.


9 months ago

The fact you managed to run it ephemeral for 2 years is crazy btw


davebecherer2003
PROOP

9 months ago

the documentation specifically mentions rollbacks for failed migrations. and thanks for all your encouraging words, your customer service skills are top notch.


9 months ago

Your migration did not fail.


davebecherer2003
PROOP

9 months ago

are you actually on staff for are you just "helping out"


9 months ago

Oh no I'm not staff


davebecherer2003
PROOP

9 months ago

exactly.


9 months ago

But that doesn't change the truth of my words lol


davebecherer2003
PROOP

9 months ago

yoiu're just a troll.


9 months ago

And staff are actively monitoring, they would correct me if they saw something I said was incorrect


davebecherer2003
PROOP

9 months ago

@Percy this should have come under the umbrella of "stateless migration" I'm thinking. It is possilbe that the data still exists at the old endpoint (legacy server). if i can get to it and mount a volume I have a chance to correct my issue.


9 months ago

Percy is a robot


9 months ago

Percy does not communicate


9 months ago

And this is not a stateless migration


9 months ago

Because your service was not stateless


9 months ago

The old deployment was destroyed as soon as the new one was created


davebecherer2003
PROOP

9 months ago

not true their documentation tells you that you can roll back.


9 months ago

rollbacks for failed migrations.


9 months ago

Railway knows if your migration failed or not


9 months ago

And the fact that your data wasn't actually saved anywhere removes any chance of even a failed deployment rollback


Loading...