server down - lost workspace from former employee
Anonymous
PROOP

9 months ago

a former employee was removed and i lost access to the workspace. as a result server is now down. i contact the former employee and when he logs in he no longer sees the workspace either.

Solved

42 Replies

Anonymous
PROOP

9 months ago

project is not visible so no project id


Anonymous
PROOP

9 months ago

trying to get this sorted asap as our server for our shopify site is currently down


adam
MODERATOR

9 months ago

Project ID is applicable here. If you don’t have it recorded, check your browser history


adam
MODERATOR

9 months ago

If the project was deleted (“former employee” suggests malicious action) the team cannot restore your project.


Anonymous
PROOP

9 months ago

ok i'll see if i can get the project id.


Anonymous
PROOP

9 months ago

the project shouldn't have been deleted. he invited my account to the workspace as admin to be able to take it over and transfer it to our team. he was removed from the admin though.


adam
MODERATOR

9 months ago

if the transfer wasn’t completed, they may have been able to delete the project.


Anonymous
PROOP

9 months ago

this was the invite that i received to join the project https://railway.com/team/invite/cDNp6fOFXK6QEsOXMKo0l



Anonymous
PROOP

9 months ago

i have to take his word for it but he said he didn't delete it and when he logs in now he doesn't see it either


adam
MODERATOR

9 months ago

Will have to flag this to the team, you’ll need to be patient as they are currently on a retreat.


adam
MODERATOR

9 months ago

!t


adam
MODERATOR

9 months ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response adam 10 months ago


Anonymous
PROOP

9 months ago

ok thanks


Anonymous
PROOP

9 months ago

afaik nobody has deleted anything but nobody has access to it


Anonymous
PROOP

9 months ago

we could probably rebuild ground up in a few days but most important is the existing db


brody
EMPLOYEE

9 months ago

going to need a project Id here, unfortunately an invite id is not super useful


Anonymous
PROOP

9 months ago

i'm working to see if that was documented anywhere. is there any way to see projects that i recently had access to but no longer. i was able to view it earlier today and had access. if that was attached to my name/account in anyway that you could help me uncover.


brody
EMPLOYEE

9 months ago

Adam suggested to look in your browser history



brody
EMPLOYEE

9 months ago

that has been deleted


Anonymous
PROOP

9 months ago

How did it get deleted? If we only changed admins and removed the former employees access? There’s no way to recover the db?


brody
EMPLOYEE

9 months ago

someone with admin access to the project deleted it


Anonymous
PROOP

9 months ago

damn. no grace period. we might be sol but that db has promo codes and gift certificate codes for customers and the record for who was distributed which codes. there's no way to recover any of that data?


brody
EMPLOYEE

9 months ago

We do have a 48 hour grace period, but it's either been 48 hours, or they went back and force deleted it before the grace period ended.


Anonymous
PROOP

9 months ago

i had admin access along with ken (former employee) i removed his admin access but that was all. i didn't delete. are you able to confirm which admin account deleted it? if it was malicious by the former employee then that needs to be known.


Anonymous
PROOP

9 months ago

i'll double-confirm with him in case that was a different project and a different project id exists for the project in question.


brody
EMPLOYEE

9 months ago

My co-worker has let me know that they are going to try to get you more information here, but there's no way we can determine the reason it was deleted, we don't ask for a reason in the delete dialog.


Anonymous
PROOP

9 months ago

much appreciate. hopefully the investigation on your end is fruitful. there were even if we have to rebuild, recovering the data for our existing customers is critical. it hasn't been 48 hours since we just made the account changes today but there was a lot of back-and-forth in the process and hopefully it didn't bug out. we had to add a new user, update the payment details as a card had expired, upgrade the account from hobby so we could add users, etc. happy to go into more detail but for now i just hope something can be salvaged.


brody
EMPLOYEE

9 months ago

I'm sorry if I've given you false hope with my previous message, no matter the information he finds, we will not be able to recover anything.


brody
EMPLOYEE

9 months ago

Given this is a user initiated deletion.


Anonymous
PROOP

9 months ago

brutal. ok i hope we can just figure out what happened then. for my own sanity the rundown went as follows: i was invited to the workspace by the former employee. i was added as an admin. everything looked good. the former employees company account was deactivated and his personal account was still active (kenstclair@gmail.com). once everything had been accounted for his admin access was removed. when i went back to railway there was no workspace or project visible anymore. i called him and him check his account and he said he could no longer see the workspace or project either. that's when i started the support thread.


Anonymous
PROOP

9 months ago

no idea what happened. i read that in order to delete you have to go to the danger tab. i never did that but curious if he did. he has said he did no such thing.


Anonymous
PROOP

9 months ago

the user initiated deletion confirmation is pretty cumbersome (a good thing!) and i know i never went through that flow to delete.


Anonymous
PROOP

9 months ago

would an account deletion delete the project? i don't think any accounts were deleted necesssarily but i also don't know for sure.


brody
EMPLOYEE

9 months ago

Yes an account deletion or Workspace deletion would delete the project, as me and my co-worker look at the data, this is more likely what happened instead of a direct project delete, though either way it's still user initiated so we cannot restore anything.


Anonymous
PROOP

9 months ago

And there are no backups or anything even that are old or outdated?


brody
EMPLOYEE

9 months ago

truthfully, yes, but per our internal polices, we cannot restore data for user initiated deletions, the backups we hold are for our own use in the event of some platform failure


Anonymous
PROOP

9 months ago

we are a startup. we just went through layoffs which is why i'm working with a former employee on this. the user initiated deletion was unintended and we're scrambling to make these handoffs without all the wheels coming off. if there's any way to petition a backup i'd like to do so just to serve the customers we currently have that we no longer have a mapping for.


brody
EMPLOYEE

9 months ago

I'm really sorry but we cannot accommodate that request, i was truthful with you because there is no point in fibbing here, so please accept that we cannot break policy for this case.


echohack
EMPLOYEE

9 months ago

Hey there Dominic,

Brody and I looked at this together and determined that the project e7339f21-3706-4dea-94bf-0328a2f60a61 was deleted around2025-04-09 UTC 15:19 We also confirmed that 4 associated deployments, volumes, and anything else in the project along with the environment associated with this project were deleted.

Unfortunately because the manner this was deleted by the Admin user, the data is unrecoverable and there is nothing further we can do to help recover it.


Status changed to Awaiting User Response Railway 10 months ago


Anonymous
PROOP

9 months ago

I really appreciate the support you provided today


Status changed to Awaiting Railway Response Railway 10 months ago


Status changed to Solved Anonymous 10 months ago


adam
MODERATOR

9 months ago

!s


Loading...