Are predeploys broken?
jasperchess
PROOP

7 days ago

I have deployments hanging exclusively on services with pre-deploy scripts, no error logs.

This particular service usually deploys in ~2minutes, but its been hanging for 20 minutes (after successfully running the pre-deploy)

Solved

9 Replies

jasperchess
PROOP

7 days ago

I temporarily removed the predeploy script and the service is now deploying correctly 👀


medim
MODERATOR

7 days ago

What type of script are you running on pre-deploy? migrations?


medim
MODERATOR

7 days ago

Mind sharing that service link?



jasperchess
PROOP

7 days ago

Yeah, migrations


jasperchess
PROOP

7 days ago

The migrations complete though, and then it just hangs

1444347371134648300


jasperchess
PROOP

7 days ago

(the "stopping container" only happened after I aborted the deploy)


medim
MODERATOR

7 days ago

🤔


medim
MODERATOR

7 days ago

Any recent changes to the pre-deploy command? Was it running fine before?


jasperchess
PROOP

7 days ago

No changes, I don't even have new migrations 😅


jasperchess
PROOP

7 days ago

Yeah been running fine for months


medim
MODERATOR

7 days ago

!t


medim
MODERATOR

7 days ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response medim 7 days ago


brody
EMPLOYEE

7 days ago

I'm not seeing anything that is hanging, could I get a link to that?



jasperchess
PROOP

7 days ago

It sat for 20 minutes before I removed the predeploy command and redeployed


jasperchess
PROOP

7 days ago


jasperchess
PROOP

7 days ago

For context, this is the script, so if the Done Migrations line is hit, there is nothing more

1444352951232499700


jasperchess
PROOP

7 days ago

It isn't a big deal for me, I'm just messing around on my staging for now, but I just wanted to raise it incase it is a broader issue 🙂


brody
EMPLOYEE

7 days ago

Can you explicitly call os.Exit()?


Status changed to Awaiting User Response Railway 7 days ago


jasperchess
PROOP

7 days ago

Oh


jasperchess
PROOP

7 days ago

That worked


brody
EMPLOYEE

7 days ago

You made the change that fast?


jasperchess
PROOP

7 days ago

I did


jasperchess
PROOP

7 days ago


brody
EMPLOYEE

7 days ago

28 second build too, nice


jasperchess
PROOP

7 days ago

Undoing the os.Exit to double check that was it


jasperchess
PROOP

7 days ago

Yeah it hangs again. That is new behaviour then, no? I haven't updated this script in a while


brody
EMPLOYEE

7 days ago

I see two other reports, so something is up on our end.


jasperchess
PROOP

7 days ago

Ah alright, good luck! <:salute:1137099685417451530>

Thanks for the help as always <:bufo_pray:1331539894769684513>


brody
EMPLOYEE

7 days ago

No problem!


efstajas
PRO

7 days ago

sorry for the "same", but same

1444369025457983700


medim
MODERATOR

7 days ago

As suggested previously, try adding an exit command at the end of your script!


efstajas
PRO

7 days ago

i assume the previous behavior will be restored? would rather wait then, my predeploy cmd is currently just running drizzle-kit migrate. would like to avoid wrapping it in a custom script on 3 different services


efstajas
PRO

7 days ago

drizzle-kit migrate definitely exits with 0 on success btw 🤔

1444371294312333600


brody
EMPLOYEE

7 days ago

I haven't attempted to reproduce yet, and even so, it's unfortunately not something we will tackle over the weekend.


efstajas
PRO

7 days ago

Could you maybe clarify what this new requirement is for predeploy commands? I understand in the other case adding an explicit exit made it work, but my command already exits with 0 (and always has), so I don't really see what else I can do at the moment to make it work.


efstajas
PRO

7 days ago

Also, ofc understand it's a weekend, but if this new behavior is unintended and unclear, feels like this would warrant an incident? 🤔


maciejrrr
HOBBY

7 days ago

Also having problems with deployments now. Everything is stuck.


Status changed to Awaiting Railway Response Railway 7 days ago


maciejrrr

Also having problems with deployments now. Everything is stuck.

brody
EMPLOYEE

7 days ago

Please open your own thread.


Status changed to Awaiting User Response Railway 7 days ago


brody
EMPLOYEE

7 days ago

It's not a new requirement, but I have yet to confirm if it is a regression. An incident won't be called, as incidents are for platform-wide issues that affect the majority of the platform.


efstajas
PRO

7 days ago

Alrighty. Re it being a regression - I can say in our case no changes have been made to anything pre-deploy related since our last successful deploy for this same service on Friday at Nov 28, 2025, 3:58:37 PM GMT+1. Today since (at the latest) Nov 29, 2025, 5:26:24 PM GMT+1 our deploys are stuck.


brody
EMPLOYEE

7 days ago

@json - Could I ask you to open your own thread? it makes it easier for us to track.


Railway
BOT

7 days ago

Hello!

We're acknowledging your issue and attaching a ticket to this thread.

We don't have an ETA for it, but, our engineering team will take a look and you will be updated as we update the ticket.

Please reply to this thread if you have any questions!


Railway
BOT

7 days ago

🛠️ The ticket Pre-deployment delays occurring has been marked as triage.


Railway
BOT

7 days ago

🛠️ The ticket Deployment delays occurring has been marked as triage.


Railway
BOT

6 days ago

🛠️ The ticket Deployment delays occurring has been marked as in progress.


brody
EMPLOYEE

6 days ago

Update here: While we have not tackled the root cause, we have fixed it for the time being and will address the root cause this week. Your pre-deploy commands should now finish just as they always have.


Status changed to Solved medim 5 days ago


Railway
BOT

3 days ago

✅ The ticket Deployment delays occurring has been marked as completed.


Loading...