Deployments are stuck for a while now
devcsrj
PROOP

2 years ago

In my project, the deployments of my services have been stuck for a while now:

  • 027b6ff3-a5b1-4c74-a272-46b5966ad7fa - 56mins+

  • 266ed302-8c1d-4774-9099-a9a202a7a090 - 42mins+

  • 9bd9f4de-51e6-40ec-99be-2a03fc4fb0a8 - 12mins+

In their logs, all of them are stuck similarly:

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 15.73MB / 43.52MB 3403.1s
#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 15.73MB / 43.52MB 3408.3s
#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 15.73MB / 43.52MB 3413.4s
#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 15.73MB / 43.52MB 3418.5s
#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 15.73MB / 43.52MB 3423.5s

Is there outage on the builders?

Solved

11 Replies

2 years ago

Hey! We were not experiencing any issues at this time, no. It seems like everything has deployed since your reported this issue. Are you still seeing slow build/deploys?


devcsrj
PROOP

2 years ago

Yeah, looks like all the deployments have finished now.

You may close this ticket.

However, would you be able to escalate on this to figure out why this happened? The common thing about these services was that I upgraded all of their npm dependencies. In the past, I also experienced a weird behavior with railway when I upgrade my dependencies. Any commits I push after the dependency upgrade commit doesn't get reflected for a few hours(?) even if the deployments were successful. I suspect these two events might have a similar root cause.

--

Here's the URLs of said timed-out builds:


2 years ago

If you see that issue with dependencies not getting rolled out please open a ticket and we can look into it

Going to close as the original issue seems resolved and new deploys seem to be going out just fine


devcsrj
PROOP

2 years ago

It happened again, here's another example. This one is just normal commit, no new npm dependencies. Now the builds are OK again now. This issue seems to be happening intermittently. Similar logs:

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3547.3s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3552.4s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3557.4s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3562.5s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3567.6s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3572.7s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3577.8s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3582.9s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3588.0s

 

#5 sha256:4ca66ffdb79362fd7157174db640d24cff5594d6acc3d12193766a2fc158c2cc 40.89MB / 43.52MB 3593.1s

2 years ago

Do you by chance know how large your image is?


2 years ago

When did this occur? Railway has an incident earlier -

https://status.railway.app/clxkpme3f722914e5osb5r304gf


devcsrj
PROOP

2 years ago

Do you by chance know how large your image is?

Around 114MB~ ish

When did this occur?

Yesterday, June 18 1500~ GMT+8. So way before that incident

My original posts meanwhile the issue happened at June 17 2248~ GMT+8


2 years ago

Do you have any idea when this might occur? It seems at least from inspection to be random?


devcsrj
PROOP

2 years ago

Timing wise, it seems random. Couldn't figure out yet why. I'll keep updating this thread for when I encounter it again.

Might be unrelated, but I also tried using the v2 deployer on a subset of my services (i.e.: some legacy, some v2). In my original post, 2 was using legacy, 1 was v2 but all 3 was affected by the build timeout.


2 years ago

Build and runtime have no relation/overlap


2 years ago

Unfortunately at this point we can't diagnose this issue and it seems to only occur for your application

I've created a ticket for this internally on the backlog. If we get more issues, we will circle back here


Railway
BOT

7 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 7 months ago


Loading...