16 days ago
Looking for an infra engineer with volume-snapshot access.
Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca
Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01
Volume: vol_1gd0898d69jh8a7u
Old host: 5e550b77-bfb3-41ef-8f64-8bbdc4ae5893 (2 CPU, 1 GB)
New host: 48273219-7fdd-4cf3-8cf9-52c90d3e39e7 (24 CPU, 24 GB)
Timeline:
Deployment bf54ef76 ran on the old host from 2026-03-28 until shutdown at 2026-05-06 19:01:37 UTC with all my prod data on vol_1gd0898d69jh8a7u.
I upgraded the plan.
5 seconds later, deployment 233890cd started on the new host at 19:01:42 UTC, mounting the same volume id β but it came up effectively empty.
Proof it's a fresh volume on the new host (not corruption):
Every table in the railway DB shows CREATE_TIME of 2026-05-06 18:30β18:47 UTC. Schema present, 0 rows in 32 of 34 business tables (only InstantlyOutreachMailbox seed + _prisma_migrations survived). mysql-volume shows 189 MB used (system + schema, no rows).
Conclusion: cross-host volume migration during plan upgrade didn't propagate data from the old host's snapshot to the new host. Data must still be in the old host's storage.
Ask:
Locate the snapshot of vol_1gd0898d69jh8a7u on host 5e550b77-... as of 2026-05-06 19:01:37 UTC (immediately before SHUTDOWN).
Reattach to MySQL OR give me read-only access to mysqldump it.
Reserve the snapshot from grace-period cleanup while we work this.
Already ruled out: manual backup at 19:01 UTC was post-wipe (177 MB schema only, 0 rows). All 5 detached volumes in the project show 0 MB.
Full CLI / SQL output ready to share. DMs open. π
13 Replies
16 days ago
There is no such thing as a cross-host volume migrate on plan upgrade, nothing ever moved hosts.
Thanks for the correction β got it, no host migration. But the data is still gone, and I need help understanding what could have caused this on the same volume.
Empirical facts I can show:
Volume id stayed the same through the whole sequence (vol_1gd0898d69jh8a7u).
But CREATE_TIME on every table in the railway DB is from 2026-05-06 18:30:09 β 18:47:32 UTC, all clustered today. If the volume's data files were preserved, those CREATE_TIME values should be from when the schema was first set up months/weeks ago β not from today.
The original deployment (bf54ef76) was still running and serving data right up until SHUTDOWN at 19:01:37 UTC. So whatever wiped the tables happened during that running deployment, ~30 min before shutdown.
Five "Restore from backup" attempts created 5 detached volumes, all 0 MB. The 19:01 manual backup itself is 177 MB but contains no row data (we mounted and verified β only schema + 6 InstantlyOutreachMailbox seed rows + _prisma_migrations).
So my new question is: on the same volume, what could rewrite every table's CREATE_TIME to a 17-minute window today, while leaving the volume id intact?
A Restore from backup operation? (Would explain the timestamps if Railway's restore does a DROP TABLE / re-import even when the backup turns out to be empty.)
A volume reset triggered somewhere in the deploy/upgrade flow?
Something on the user side I missed?
If the wipe was triggered by a Railway-side restore operation against an empty backup β is there any point-in-time recovery, hourly snapshot, replication, or backend-side history of vol_1gd0898d69jh8a7u from BEFORE today's 18:30 UTC mark that I could recover from? Even a stale daily snapshot would be vastly better than nothing.
Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca
Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01
Volume: vol_1gd0898d69jh8a7u
16 days ago
This was something the user has done, such as a forced migration, a manual volume wipe, etc.
Understood β you're saying it was user-triggered. I won't push back on that since the practical question is the same either way:
Do you have any point-in-time snapshots or automatic daily backups of volume vol_1gd0898d69jh8a7u from BEFORE 2026-05-06 18:30 UTC? Even a 24-hour-stale snapshot is vastly better than starting from zero.
If the answer is "no, only manual backups exist and yours was post-wipe" β I'll accept that and stop bothering you. Just need a definitive yes/no on backend snapshots so I know whether to wait or move on to a full rebuild from external sources.
Project: endearing-grace Β· Service 6637ec33-d5a4-4a2f-961b-49fc0b052e01 Β· Volume vol_1gd0898d69jh8a7u
16 days ago
No, only manual backups exist, and yours was post-wipe.
so if user upgrade the plan and they have to redeploy the service so you are saying thats users issue right?
16 days ago
The redeploy did not cause the data loss; the data loss was caused by something else the user did.
16 days ago
The data loss was caused by something else the user did. The redeploy did not cause the data loss.
Attachments
16 days ago
I'll be disengaging now. Sorry we aren't able to help you further.