9 months ago
Project- CNTG-VIDEORECOMMENDER-BASE
Issue: upgrade to Railway Metal completely reinitialized my MYSQL database. My tables are gone, which represent about 2 years of work.
Project ID: 421de913-10a8-48a4-9c91-a736510d543d
Environment ID: 39809c9a-bceb-4019-a943-dcc01d4e9ba0
Is it possible that there is a backup on your end and a recovery team that can please assist?
<@&743878049736687688> Team - im reading the guide ansee that it is possible that hte migration is still underway?
30 Replies
@Percy - Thank you for acknowledging this!
Can you confirm:
Is the Metal migration still in progress for my database?
Is my production data being transferred in the background?
This is affecting my live application with paying customers.
9 months ago
You obviously had this data on a volume, right?
9 months ago
Redeployment was never an issue?
9 months ago
To be clear, migration is not still underway. It has completed for your database.
@Loudbook
i have a volume, that looks empty. it may have never been attached. This may be my problem. I set this up 2 years ago when I knew very little about it. I've been iterating and building on it since then. I do have some other places I can rebuild those tables from and this may be a terrible lesson learned for me that backups are not happening without attached volumes. I thought I had taken care of it with that holiday-volume in my project but it doesn't appear to be mounted and it seems empty.
9 months ago
If it’s unmounted and it never was, then your data is gone.
9 months ago
No backups are taken by Railway. Your database is your responsibility.
@Loudbook - I understand the volume issue for the future. Right now I'm focused on recovery options, but I'm confused about what happened based on your documentation.
Your Metal migration docs state:
"For Stateless deployments, meaning, a deployment with no volume- there is no downtime. Stateless deployments are just landing into a new region."
But what I experienced was complete database reinitialization and total data loss - which doesn't match what should happen for a stateless migration.
This discrepancy makes me wonder if there might be recovery options I'm not aware of:
Since this didn't follow the normal stateless migration process, are there any automatic snapshots or backups Railway takes during failed migrations?
Could there be any remnants in the old region that weren't properly migrated?
Are there any system logs or temporary files from the migration process that might contain data?
Does Railway keep any rollback data when migrations don't go as documented?
I'm not looking to assign blame - I just need to know if the unexpected behavior means there might be ANY possibility of data recovery before I start rebuilding from scratch.
Even partial data or table structures would be helpful at this point. in the future of course i'll make sure the volume is attached, and as much as I am to blame here for not undersatnd how this works, I'm hoping that there is SOME remedy as I've got several sites relying on these tables that are now down.
@Loudbook - One more question about rollback:
Since your docs say stateless migrations are "just landing into a new region," if I rollback to the old region, would I reconnect to my original container/data?
Or does Railway immediately destroy the old deployment when migrating?
Given that my migration didn't behave like the documented stateless process, I'm wondering if my original data might still exist in the old region.
9 months ago
There is no backup. Your data is gone. Period.
9 months ago
The fact you managed to run it ephemeral for 2 years is crazy btw
the documentation specifically mentions rollbacks for failed migrations. and thanks for all your encouraging words, your customer service skills are top notch.
9 months ago
Your migration did not fail.
9 months ago
Oh no I'm not staff
9 months ago
But that doesn't change the truth of my words lol
9 months ago
And staff are actively monitoring, they would correct me if they saw something I said was incorrect
@Percy this should have come under the umbrella of "stateless migration" I'm thinking. It is possilbe that the data still exists at the old endpoint (legacy server). if i can get to it and mount a volume I have a chance to correct my issue.
9 months ago
Percy is a robot
9 months ago
Percy does not communicate
9 months ago
And this is not a stateless migration
9 months ago
Because your service was not stateless
9 months ago
The old deployment was destroyed as soon as the new one was created
9 months ago
rollbacks for failed migrations.
9 months ago
Railway knows if your migration failed or not
9 months ago
And the fact that your data wasn't actually saved anywhere removes any chance of even a failed deployment rollback
