🚨 URGENT: Need Railway Staff Support - Data Loss from Metal Migration

davebecherer2003
PRO

2 months ago

URGENT: Data Loss from Metal Migration - Need Railway Engineering Support

Project ID: 421de913-10a8-48a4-9c91-a736510d543d

SITUATION:

My production MySQL service lost all data during Railway's forced Metal migration. This database powers multiple live customer sites that are now down.

TECHNICAL DETAILS:

- MySQL service was running successfully for 2+ years

- Database contained critical production data (user accounts, content, etc.)

- I now realize the volume wasn't properly attached (my configuration error)

- After migration: complete database reinitialization, all tables empty

DOCUMENTATION DISCREPANCY:

According to Railway's Metal migration docs, services without volumes should be treated as "stateless migrations" with:

- "No downtime"

- "Just landing into a new region"

However, I experienced complete data loss and reinitialization, which contradicts the documented stateless migration behavior.

RECOVERY REQUEST:

Your documentation mentions:

1. Backups created during migration process

2. Rollback options for failed migrations

3. Data preservation for stateless deployments

I'm attempting to rollback to legacy regions hoping the original data still exists there.

Can Railway engineering staff help me:

- Attempt the documented rollback procedures

- Check if data exists in the old region before cleanup

- Understand why this wasn't handled as a stateless migration

I accept responsibility for the volume configuration issue, but the migration behavior doesn't match your documented process.

@Railway Team - Need official engineering support only

Solved

2 Replies

Railway
BOT

2 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway 2 months ago


Railway
BOT

2 months ago

✅ The internal ticket Metal volume data loss has been marked as completed.


sarahkb125
EMPLOYEE

2 months ago

Hi there - the volume was configured such that it was ephemeral storage - the volume was not attached correctly in your setup. So, if anything were to change (even a crash or restart), the data would be lost. Unfortunately because it wasn't attached to a volume, there is no way to recover it.

I would suggest making sure the volume is properly attached so this does not happen again.


Status changed to Solved sarahkb125 2 months ago


🚨 URGENT: Need Railway Staff Support - Data Loss from Metal Migration - Railway Help Station