a month ago
Looking for an infra engineer with volume-snapshot access.
Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca
Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01
Volume: vol_1gd0898d69jh8a7u
Old host: 5e550b77-bfb3-41ef-8f64-8bbdc4ae5893 (2 CPU, 1 GB)
New host: 48273219-7fdd-4cf3-8cf9-52c90d3e39e7 (24 CPU, 24 GB)
Timeline:
Deployment bf54ef76 ran on the old host from 2026-03-28 until shutdown at 2026-05-06 19:01:37 UTC with all my prod data on vol_1gd0898d69jh8a7u.
I upgraded the plan.
5 seconds later, deployment 233890cd started on the new host at 19:01:42 UTC, mounting the same volume id β but it came up effectively empty.
Proof it's a fresh volume on the new host (not corruption):
Every table in the railway DB shows CREATE_TIME of 2026-05-06 18:30β18:47 UTC. Schema present, 0 rows in 32 of 34 business tables (only InstantlyOutreachMailbox seed + _prisma_migrations survived). mysql-volume shows 189 MB used (system + schema, no rows).
Conclusion: cross-host volume migration during plan upgrade didn't propagate data from the old host's snapshot to the new host. Data must still be in the old host's storage.
Ask:
Locate the snapshot of vol_1gd0898d69jh8a7u on host 5e550b77-... as of 2026-05-06 19:01:37 UTC (immediately before SHUTDOWN).
Reattach to MySQL OR give me read-only access to mysqldump it.
Reserve the snapshot from grace-period cleanup while we work this.
Already ruled out: manual backup at 19:01 UTC was post-wipe (177 MB schema only, 0 rows). All 5 detached volumes in the project show 0 MB.
Full CLI / SQL output ready to share. DMs open. π
13 Replies
a month ago
There is no such thing as a cross-host volume migrate on plan upgrade, nothing ever moved hosts.
Thanks for the correction β got it, no host migration. But the data is still gone, and I need help understanding what could have caused this on the same volume.
Empirical facts I can show:
Volume id stayed the same through the whole sequence (vol_1gd0898d69jh8a7u).
But CREATE_TIME on every table in the railway DB is from 2026-05-06 18:30:09 β 18:47:32 UTC, all clustered today. If the volume's data files were preserved, those CREATE_TIME values should be from when the schema was first set up months/weeks ago β not from today.
The original deployment (bf54ef76) was still running and serving data right up until SHUTDOWN at 19:01:37 UTC. So whatever wiped the tables happened during that running deployment, ~30 min before shutdown.
Five "Restore from backup" attempts created 5 detached volumes, all 0 MB. The 19:01 manual backup itself is 177 MB but contains no row data (we mounted and verified β only schema + 6 InstantlyOutreachMailbox seed rows + _prisma_migrations).
So my new question is: on the same volume, what could rewrite every table's CREATE_TIME to a 17-minute window today, while leaving the volume id intact?
A Restore from backup operation? (Would explain the timestamps if Railway's restore does a DROP TABLE / re-import even when the backup turns out to be empty.)
A volume reset triggered somewhere in the deploy/upgrade flow?
Something on the user side I missed?
If the wipe was triggered by a Railway-side restore operation against an empty backup β is there any point-in-time recovery, hourly snapshot, replication, or backend-side history of vol_1gd0898d69jh8a7u from BEFORE today's 18:30 UTC mark that I could recover from? Even a stale daily snapshot would be vastly better than nothing.
Project: endearing-grace / 822f75b8-a0bf-43e8-ae19-7c31c4d4c1ca
Service: MySQL / 6637ec33-d5a4-4a2f-961b-49fc0b052e01
Volume: vol_1gd0898d69jh8a7u
a month ago
This was something the user has done, such as a forced migration, a manual volume wipe, etc.
Understood β you're saying it was user-triggered. I won't push back on that since the practical question is the same either way:
Do you have any point-in-time snapshots or automatic daily backups of volume vol_1gd0898d69jh8a7u from BEFORE 2026-05-06 18:30 UTC? Even a 24-hour-stale snapshot is vastly better than starting from zero.
If the answer is "no, only manual backups exist and yours was post-wipe" β I'll accept that and stop bothering you. Just need a definitive yes/no on backend snapshots so I know whether to wait or move on to a full rebuild from external sources.
Project: endearing-grace Β· Service 6637ec33-d5a4-4a2f-961b-49fc0b052e01 Β· Volume vol_1gd0898d69jh8a7u
a month ago
No, only manual backups exist, and yours was post-wipe.
so if user upgrade the plan and they have to redeploy the service so you are saying thats users issue right?
a month ago
The redeploy did not cause the data loss; the data loss was caused by something else the user did.
a month ago
The data loss was caused by something else the user did. The redeploy did not cause the data loss.
Attachments
a month ago
I'll be disengaging now. Sorry we aren't able to help you further.