9 months ago
a former employee was removed and i lost access to the workspace. as a result server is now down. i contact the former employee and when he logs in he no longer sees the workspace either.
42 Replies
trying to get this sorted asap as our server for our shopify site is currently down
9 months ago
Project ID is applicable here. If you don’t have it recorded, check your browser history
9 months ago
If the project was deleted (“former employee” suggests malicious action) the team cannot restore your project.
the project shouldn't have been deleted. he invited my account to the workspace as admin to be able to take it over and transfer it to our team. he was removed from the admin though.
9 months ago
if the transfer wasn’t completed, they may have been able to delete the project.
this was the invite that i received to join the project https://railway.com/team/invite/cDNp6fOFXK6QEsOXMKo0l
i have to take his word for it but he said he didn't delete it and when he logs in now he doesn't see it either
9 months ago
Will have to flag this to the team, you’ll need to be patient as they are currently on a retreat.
9 months ago
!t
9 months ago
This thread has been escalated to the Railway team.
Status changed to Awaiting Railway Response adam • 10 months ago
we could probably rebuild ground up in a few days but most important is the existing db
9 months ago
going to need a project Id here, unfortunately an invite id is not super useful
9 months ago
i'm working to see if that was documented anywhere. is there any way to see projects that i recently had access to but no longer. i was able to view it earlier today and had access. if that was attached to my name/account in anyway that you could help me uncover.
9 months ago
Adam suggested to look in your browser history
9 months ago
9 months ago
that has been deleted
9 months ago
How did it get deleted? If we only changed admins and removed the former employees access? There’s no way to recover the db?
9 months ago
someone with admin access to the project deleted it
9 months ago
damn. no grace period. we might be sol but that db has promo codes and gift certificate codes for customers and the record for who was distributed which codes. there's no way to recover any of that data?
9 months ago
We do have a 48 hour grace period, but it's either been 48 hours, or they went back and force deleted it before the grace period ended.
9 months ago
i had admin access along with ken (former employee) i removed his admin access but that was all. i didn't delete. are you able to confirm which admin account deleted it? if it was malicious by the former employee then that needs to be known.
9 months ago
i'll double-confirm with him in case that was a different project and a different project id exists for the project in question.
9 months ago
My co-worker has let me know that they are going to try to get you more information here, but there's no way we can determine the reason it was deleted, we don't ask for a reason in the delete dialog.
9 months ago
much appreciate. hopefully the investigation on your end is fruitful. there were even if we have to rebuild, recovering the data for our existing customers is critical. it hasn't been 48 hours since we just made the account changes today but there was a lot of back-and-forth in the process and hopefully it didn't bug out. we had to add a new user, update the payment details as a card had expired, upgrade the account from hobby so we could add users, etc. happy to go into more detail but for now i just hope something can be salvaged.
9 months ago
I'm sorry if I've given you false hope with my previous message, no matter the information he finds, we will not be able to recover anything.
9 months ago
Given this is a user initiated deletion.
9 months ago
brutal. ok i hope we can just figure out what happened then. for my own sanity the rundown went as follows: i was invited to the workspace by the former employee. i was added as an admin. everything looked good. the former employees company account was deactivated and his personal account was still active (kenstclair@gmail.com). once everything had been accounted for his admin access was removed. when i went back to railway there was no workspace or project visible anymore. i called him and him check his account and he said he could no longer see the workspace or project either. that's when i started the support thread.
9 months ago
no idea what happened. i read that in order to delete you have to go to the danger tab. i never did that but curious if he did. he has said he did no such thing.
9 months ago
the user initiated deletion confirmation is pretty cumbersome (a good thing!) and i know i never went through that flow to delete.
9 months ago
would an account deletion delete the project? i don't think any accounts were deleted necesssarily but i also don't know for sure.
9 months ago
Yes an account deletion or Workspace deletion would delete the project, as me and my co-worker look at the data, this is more likely what happened instead of a direct project delete, though either way it's still user initiated so we cannot restore anything.
9 months ago
And there are no backups or anything even that are old or outdated?
9 months ago
truthfully, yes, but per our internal polices, we cannot restore data for user initiated deletions, the backups we hold are for our own use in the event of some platform failure
9 months ago
we are a startup. we just went through layoffs which is why i'm working with a former employee on this. the user initiated deletion was unintended and we're scrambling to make these handoffs without all the wheels coming off. if there's any way to petition a backup i'd like to do so just to serve the customers we currently have that we no longer have a mapping for.
9 months ago
I'm really sorry but we cannot accommodate that request, i was truthful with you because there is no point in fibbing here, so please accept that we cannot break policy for this case.
9 months ago
Hey there Dominic,
Brody and I looked at this together and determined that the project e7339f21-3706-4dea-94bf-0328a2f60a61 was deleted around2025-04-09 UTC 15:19 We also confirmed that 4 associated deployments, volumes, and anything else in the project along with the environment associated with this project were deleted.
Unfortunately because the manner this was deleted by the Admin user, the data is unrecoverable and there is nothing further we can do to help recover it.
Status changed to Awaiting User Response Railway • 10 months ago
9 months ago
I really appreciate the support you provided today
Status changed to Awaiting Railway Response Railway • 10 months ago
Status changed to Solved Anonymous • 10 months ago
9 months ago
!s