11 hours ago
Per the title: Getting more things failing in build steps, which succeed after one or more retries. These all require manual intervention, which defeats the purpose of the automation.
Even just a simple "backoff and redeploy" feature would take my complaints from a few a week to 0. The current automatic retries don't work when it fails in the build step.
I could configure an automatic retry myself, but that defeats part of the purpose for why I and others use railway. It's supposed to give me less stuff to worry about, not more.
I'm also pretty sure I've made multiple posts here and in Discord about intermittent failures, but it seems to be a problem continually unaddressed by Railway. Many others report similar things in Discord.
Linked a sample failed service. (which is now succeeding after two manual retries)
5 Replies
Status changed to Awaiting Railway Response Railway • about 11 hours ago
11 hours ago
This is on our side. There is an active incident on our build infrastructure in US East and EU West (https://status.railway.com/incident/TX3M5SWF): some builds are failing before any of your build steps run, with the build daemon connection dropping or the build machine never coming up, which is exactly what your deployments show. The cause has been identified and a fix is being worked on with the upstream provider. Your source, build configuration and dependencies are not at fault, and nothing needs to change on your end.
Running services are not affected. A fresh deployment of the same commit may get through in the meantime; otherwise please follow the status page for the all-clear and redeploy once it is posted.
Status changed to Awaiting User Response Railway • about 11 hours ago
dizzydes90
This is on our side. There is an active incident on our build infrastructure in US East and EU West (https://status.railway.com/incident/TX3M5SWF): some builds are failing before any of your build steps run, with the build daemon connection dropping or the build machine never coming up, which is exactly what your deployments show. The cause has been identified and a fix is being worked on with the upstream provider. Your source, build configuration and dependencies are not at fault, and nothing needs to change on your end. Running services are not affected. A fresh deployment of the same commit may get through in the meantime; otherwise please follow the status page for the all-clear and redeploy once it is posted.
9 hours ago
Hi, I am speaking of more than just today. If you check that same linked deployment for some examples, this has happened randomly over the last 5 or so days. This is something more than just the current temporary outage.
Status changed to Awaiting Railway Response Railway • about 9 hours ago
9 hours ago
That's fair, and our earlier reply only covered today's incident, which has since been resolved. It doesn't account for failures from the days before it.
If a build fails again from here, please share the deployment link and roughly when it happened, so we can look at that specific failure.
Thanks for the suggestion on automatic backoff and redeploy for failed builds. Right now, a build that fails in the build step has to be redeployed manually.
Status changed to Awaiting User Response Railway • about 9 hours ago
8 hours ago
Two that just failed:
related, the "Please check the build logs for more details." link has never worked for me -- it just appends #build to the url, but doesn't change the current view
Status changed to Awaiting Railway Response Railway • about 8 hours ago
5 hours ago
Sorry you ran into more failures after our earlier all-clear. The transient upstream-provider issue has since been resolved, and the latest failures should now be cleared up.
We’re continuing to improve build reliability and address the intermittent failures you’ve described over the past several days. Thanks also for flagging the build-log link failing to open the correct view.
Status changed to Awaiting User Response Railway • about 5 hours ago