24 days ago
This seems to come up often in railway support threads (here, here, and others), but the advice is always just "redeploy" (literally not an option for failed deployments in the UI) or "push another commit" (requiring a full change -> review -> staging -> release cycle).
The "restart policy" causes it to just do this same failure for the number of retries.
It's not like this is happening every day, but when it does during production deploys, it causes an outage for services don't have a healthcheck configured (that one is on me, workers need healthchecks too).
Can we get a more permanent fix for this issue?
9 Replies
Status changed to Awaiting Railway Response Railway • 24 days ago
24 days ago
This is a transient build failure that resolves on its own - a subsequent deploy of the same code will succeed. We understand it's disruptive when it hits a production deploy.
You're right that healthchecks are the key protection here. With a healthcheck configured, a failed deploy never replaces the running deployment, so the production impact goes away. Adding healthchecks to your worker services will close that gap.
For retriggering a deploy after a failure, you can use the three-dot menu on the most recent successful deployment to redeploy from there. Your broader point about wanting this to not happen in the first place is well taken.
Status changed to Awaiting User Response Railway • 24 days ago
sam-a
This is a transient build failure that resolves on its own - a subsequent deploy of the same code will succeed. We understand it's disruptive when it hits a production deploy. You're right that healthchecks are the key protection here. With a healthcheck configured, a failed deploy never replaces the running deployment, so the production impact goes away. Adding healthchecks to your worker services will close that gap. For retriggering a deploy after a failure, you can use the three-dot menu on the most recent successful deployment to redeploy from there. Your broader point about wanting this to not happen in the first place is well taken.
24 days ago
Actually, I dug into this service. It has a healthcheck. But also, this failure was during the build step, not deploy and yet it still removed the current deployment.
Status changed to Awaiting Railway Response Railway • 24 days ago
24 days ago
This issue has been occurring for me all afternoon, too. Contrary to the assertion that a redeploy will clear it up, Whether I manually redeploy via the dot menu or push an update to git that triggers the redeploy, the failed to fetch snapshot issue persists.
Reference - Railway Service ID 706cf7fe-d6b6-4931-a494-5c23f6c88145
24 days ago
We tracked this down. When a build is skipped because nothing matched your watch patterns, the source snapshot can get recorded before its upload finishes, so your next commit deploy fails looking for it rather than falling back to a full build. Retrying the deploy clears it, no new commit needed. If you hit it straight after a Redeploy instead, deploy fresh from source, since Redeploy points back at the same snapshot. Your previous deployment did keep serving throughout, it wasn't removed until your successful redeploy went live nearly three hours later.
Status changed to Awaiting User Response Railway • 24 days ago
brody
We tracked this down. When a build is skipped because nothing matched your watch patterns, the source snapshot can get recorded before its upload finishes, so your next commit deploy fails looking for it rather than falling back to a full build. Retrying the deploy clears it, no new commit needed. If you hit it straight after a Redeploy instead, deploy fresh from source, since Redeploy points back at the same snapshot. Your previous deployment did keep serving throughout, it wasn't removed until your successful redeploy went live nearly three hours later.
24 days ago
Just in case I didn't link it above (can't tell), here's my service and the failed deploy: https://railway.com/project/7f6df885-6c07-4891-accd-09bbcb441429/service/4e7ba749-2707-4e3c-b53e-7181251bd39a?groupId=96fd4100-96e0-4d2f-9e2f-e90ebdfc5274&environmentId=aef3c71b-ccb7-43b5-a6ed-cfca5f9a50d2&id=857d0acd-69d1-4024-94b0-7c06d596cd8b
My service did not keep serving throughput as far as I can tell. I tried accessing it during this to see and landed on the Railway "your application is not responding" page. I ignored the initial alert because I (incorrectly) assumed it would keep serving since it was just a build failure. After opening it and finding it broken, I clicked around trying to get it back up until I pushed another change. That's what drove me to post, it was counter to expected behavior.
And to reiterate, there is no redeploy option when this kind of failure happens:
Attachments
Status changed to Awaiting Railway Response Railway • 24 days ago
Status changed to Awaiting User Response Railway • 24 days ago
brody
Do we have consent to view your project?
24 days ago
yes
Status changed to Awaiting Railway Response Railway • 24 days ago
24 days ago
You currently have an active deployment that succeeded after the linked failed deployment. My understanding is that you are unblocked. Is that not the case?
Status changed to Awaiting User Response Railway • 24 days ago
24 days ago
Yes, I am unblocked. Not a P0 anymore, just something that happens periodically and is unexpected when it does.
Status changed to Awaiting Railway Response Railway • 24 days ago
Status changed to Solved Railway • 24 days ago