P0: Cross-host volume migration data loss after plan upgrade
rainmaker255
PROOP

a month ago

Looking for an infra engineer with volume-snapshot access.

Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca

Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01

Volume: vol_1gd0898d69jh8a7u

Old host: 5e550b77-bfb3-41ef-8f64-8bbdc4ae5893 (2 CPU, 1 GB)

New host: 48273219-7fdd-4cf3-8cf9-52c90d3e39e7 (24 CPU, 24 GB)

Timeline:

Deployment bf54ef76 ran on the old host from 2026-03-28 until shutdown at 2026-05-06 19:01:37 UTC with all my prod data on vol_1gd0898d69jh8a7u.

I upgraded the plan.

5 seconds later, deployment 233890cd started on the new host at 19:01:42 UTC, mounting the same volume id β€” but it came up effectively empty.

Proof it's a fresh volume on the new host (not corruption):

Every table in the railway DB shows CREATE_TIME of 2026-05-06 18:30–18:47 UTC. Schema present, 0 rows in 32 of 34 business tables (only InstantlyOutreachMailbox seed + _prisma_migrations survived). mysql-volume shows 189 MB used (system + schema, no rows).

Conclusion: cross-host volume migration during plan upgrade didn't propagate data from the old host's snapshot to the new host. Data must still be in the old host's storage.

Ask:

Locate the snapshot of vol_1gd0898d69jh8a7u on host 5e550b77-... as of 2026-05-06 19:01:37 UTC (immediately before SHUTDOWN).

Reattach to MySQL OR give me read-only access to mysqldump it.

Reserve the snapshot from grace-period cleanup while we work this.

Already ruled out: manual backup at 19:01 UTC was post-wipe (177 MB schema only, 0 rows). All 5 detached volumes in the project show 0 MB.

Full CLI / SQL output ready to share. DMs open. πŸ™

13 Replies

a month ago

There is no such thing as a cross-host volume migrate on plan upgrade, nothing ever moved hosts.


rainmaker255
PROOP

a month ago

Thanks for the correction β€” got it, no host migration. But the data is still gone, and I need help understanding what could have caused this on the same volume.

Empirical facts I can show:

Volume id stayed the same through the whole sequence (vol_1gd0898d69jh8a7u).

But CREATE_TIME on every table in the railway DB is from 2026-05-06 18:30:09 β†’ 18:47:32 UTC, all clustered today. If the volume's data files were preserved, those CREATE_TIME values should be from when the schema was first set up months/weeks ago β€” not from today.

The original deployment (bf54ef76) was still running and serving data right up until SHUTDOWN at 19:01:37 UTC. So whatever wiped the tables happened during that running deployment, ~30 min before shutdown.

Five "Restore from backup" attempts created 5 detached volumes, all 0 MB. The 19:01 manual backup itself is 177 MB but contains no row data (we mounted and verified β€” only schema + 6 InstantlyOutreachMailbox seed rows + _prisma_migrations).

So my new question is: on the same volume, what could rewrite every table's CREATE_TIME to a 17-minute window today, while leaving the volume id intact?

A Restore from backup operation? (Would explain the timestamps if Railway's restore does a DROP TABLE / re-import even when the backup turns out to be empty.)

A volume reset triggered somewhere in the deploy/upgrade flow?

Something on the user side I missed?

If the wipe was triggered by a Railway-side restore operation against an empty backup β€” is there any point-in-time recovery, hourly snapshot, replication, or backend-side history of vol_1gd0898d69jh8a7u from BEFORE today's 18:30 UTC mark that I could recover from? Even a stale daily snapshot would be vastly better than nothing.

Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca

Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01

Volume: vol_1gd0898d69jh8a7u


a month ago

This was something the user has done, such as a forced migration, a manual volume wipe, etc.


rainmaker255
PROOP

a month ago

Understood β€” you're saying it was user-triggered. I won't push back on that since the practical question is the same either way:

Do you have any point-in-time snapshots or automatic daily backups of volume vol_1gd0898d69jh8a7u from BEFORE 2026-05-06 18:30 UTC? Even a 24-hour-stale snapshot is vastly better than starting from zero.

If the answer is "no, only manual backups exist and yours was post-wipe" β€” I'll accept that and stop bothering you. Just need a definitive yes/no on backend snapshots so I know whether to wait or move on to a full rebuild from external sources.

Project: endearing-grace Β· Service 6637ec33-d5a4-4a2f-961b-49fc0b052e01 Β· Volume vol_1gd0898d69jh8a7u


a month ago

No, only manual backups exist, and yours was post-wipe.


rainmaker255
PROOP

a month ago

so if user upgrade the plan and they have to redeploy the service so you are saying thats users issue right?


a month ago

The redeploy did not cause the data loss; the data loss was caused by something else the user did.


rainmaker255
PROOP

a month ago

becasue even after upgrade that service wass not working


rainmaker255
PROOP

a month ago

it did with my service


a month ago

The data loss was caused by something else the user did. The redeploy did not cause the data loss.


rainmaker255
PROOP

a month ago

do i show you logs from CLI ??


rainmaker255
PROOP

a month ago

image.png

Attachments


a month ago

I'll be disengaging now. Sorry we aren't able to help you further.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...