Rollbacks get stuck on pre-deploy command
martinjrp
PROOP

2 months ago

I've noticed this before but it looks like when I attempt to rollback to a prior-deploy, the deployment gets stuck saying Deployment in progress: Running pre-deploy command

According to the deploy logs, the pre-deploy script seems to run successfully with the last log message being Exited with code 0.

The script in question runs 4 commands, all related to database migrations. Additionally, this is never a problem with regular deployments which does lead me to wonder if this is a bug with Rollbacks

Solved

4 Replies

Railway
BOT

2 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


echohack
EMPLOYEE

2 months ago

It sounds like an issue with your migration script sending SIGINT to the container when it exits on a redeploy. I'd expect migrations not to run on a redeploy, but rollbacks of stateful workloads can be a little tricky.

```
@headlinersapp/common db:migrate: Exited with code 0

Stopping Container
```

I don't think there's enough logging here to understand what is happening during your db:migrate step, but something is clearly breaking. Have you added more tracing to see what's going on with your migrations?


Status changed to Awaiting User Response Railway 2 months ago


Railway
BOT

2 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 2 months ago


martinjrp
PROOP

a month ago

I think I'm still running into this issue...

I've been keeping an eye on this with additional logs and guards. From what I gather, my guess is that the database connection wasn't closing so the process wouldn't quit. I've since added an explicit exit which isn't ideal but I haven't have the pre-deploy hook hang at all in regular deploys.

That said, I tried a rollback again today and same issue... hanging on the "completed" script. There were no added migrations or changes to the step between the current and target deployment. I can't tell 100% but I believe this was the first stalled deploy: 10715c3a-bea9-4098-ba21-db694d615690

I've tried deploying latest and additional commits but now the workflow seems stuck on "Waiting for previous deployment". There aren't any pending deploys.

Any assistance with getting the latest commit deployed would be appreciated -- production is down so I'd like that resolved ASAP, the rollback issue is somewhat of an aside. Thanks!


Status changed to Awaiting Railway Response Railway about 1 month ago


noahd
EMPLOYEE

a month ago

Hi there!
We identified and fixed something earlier today which might have been the cause of this. Are you still running into these issues?


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 1 month ago


Loading...