Railway incident ?? Ghost deployment without our interaction
suhjohn
PROOP

2 years ago

This is incredibly serious. We just found out that railway created a deployment out of nothing out of thin air, taking our site down, using a very old random commit.

22 Replies

suhjohn
PROOP

2 years ago

cc <@777399141746737262>


suhjohn
PROOP

2 years ago

Is it possible we got hacked, or is there something else happening here?


suhjohn
PROOP

2 years ago

120b5ec5-59d8-4087-84ae-4e0b3d934aa7


suhjohn
PROOP

2 years ago

ruling out that we got hacked, it looks like railway just triggered a ghost commit from may 2023


suhjohn
PROOP

2 years ago

since to log into railway you'd have to use magic, and that didn't happen


suhjohn
PROOP

2 years ago

so this is very messed up???


2 years ago

in order for railway to deploy your commits they listen to webhooks that github fires, it's most likely that this was github's doing


suhjohn
PROOP

2 years ago

let me check if vercel triggered


suhjohn
PROOP

2 years ago

this would quickly clarify it


2 years ago

vercel?


suhjohn
PROOP

2 years ago

if it's a github issue, both railway and vercel should be affected


suhjohn
PROOP

2 years ago

since they also use the same webhooks


suhjohn
PROOP

2 years ago

and we deploy on every commit for backend/frontend


2 years ago

not necessarily, vercel could have some additional logic in there to prevent such things


suhjohn
PROOP

2 years ago

in this case, it looks like only railway got triggered, so it narrows down to you


suhjohn
PROOP

2 years ago

Any way we can have the team look at the requests? Also wondering if there's an attack vector somewhere


suhjohn
PROOP

2 years ago

<@539512869780455445> My naive thinking at this point is that there was like a queued deployment request somehow that was stuck in a "processing" state in the db, some eng in Railway ran a script that had nothing to do with it and it got triggered


2 years ago

possible, could be a dozen causes, regardless of cause, this definitely shouldn't have happened. will flag team


2 years ago

quick question, your service is back up now right?


suhjohn
PROOP

2 years ago

Yeah! <@539512869780455445>


suhjohn
PROOP

2 years ago

we just pushed an empty commit


2 years ago

Can you link the deployment?


Loading...