8 days ago
A MySQL service with public proxy at shuttle.proxy.rlwy.net:11561 containing a database named 'css' is no longer accessible. I need to recover the data or find out if a backup exists. This is a live client site.
20 Replies
@Railway Please help with this. Client data all lost. I have a few volumes I tried remounting but it appears they have been wiped.
There was data and it was working up until a few days ago, I'm not sure exactly what caused this. I did move the project from 47-Industries git org to Vaylo-Studios, but I did not delete anything. Please help as soon as possible.
I've reseeded initial data but still missing the past 2 months of updates and data the client keyed in via admin
6 days ago
Please don't ping team members, #🛂|readme 5)
6 days ago
(no markdown available for this content)
6 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 6 days ago
6 days ago
We don't offer data recovery for user-initiated actions, so I've opened this to the community to see if they are able to offer any assistance.
My apologies, I saw this user was added to the thread on the side of my Discord so i thought they were assigned to it or something... @ to mention is ok?
5 days ago
@ to mention is not desired, Chandrika was just curious.
Either way, our apologies but we cannot assist further from our end, you can see the actions taken via your project activity logs and audit logs if available.
I truly don't understand what I as a user did I never deleted this database, i just migrated the service
5 days ago
I'm sorry, but we cannot assist with user initiated data loss, I wish we could do more here.
Even if it's not helping recover there's nothing you guys can help with to help me understand HOW it happened so I can prevent it from happening in the future? What I'm confused about is how it was user-initiated.
5 days ago
Mentioned that previously.
Hey, I want to follow up with some updated info. When I first reported this I thought the git repo migration from 47-Industries to Vaylo-Studios was the cause, that's why I mentioned it. But after going through the logs that's not what happened.
Errors started June 24th at 9:52 AM EDT. The git migration didn't happen until June 27th, three days later. So that had nothing to do with it.
What the logs show:
- June 24 9:52 AM: Prisma DB errors start, MySQL unreachable
- No deletion event in project activity logs or audit logs for that date
- No user action at all on June 24
- MySQL service disappeared completely, volume stayed behind
- June 27: git migration, unrelated
The service didn't crash, it just disappeared with no log entry. A crash would still show up in the dashboard. The volume surviving without the service is not something I did.
After finding this I tried multiple times to spin up a new MySQL and remount the old volume to get the data back. Every attempt failed with InnoDB redo log errors. The volume data was already corrupted before I touched it. I couldn't recover anything from it.
I pointed to the git migration originally because I was looking for an explanation. The timestamps say otherwise. I'd like this escalated because "user-initiated" is not what the evidence shows, and I have a client who lost real data over this.
5 days ago
I'm sorry but we cannot help with user initiated data loss
I just explained that it wasn't user initiated. The errors started June 24. I didn't do anything on June 24. The git migration was June 27. There is no deletion event in any of my logs. How is that user initiated?