failed to fetch snapshot
therobotcarlson
PROOP

24 days ago

This seems to come up often in railway support threads (here, here, and others), but the advice is always just "redeploy" (literally not an option for failed deployments in the UI) or "push another commit" (requiring a full change -> review -> staging -> release cycle).

The "restart policy" causes it to just do this same failure for the number of retries.

It's not like this is happening every day, but when it does during production deploys, it causes an outage for services don't have a healthcheck configured (that one is on me, workers need healthchecks too).

Can we get a more permanent fix for this issue?

image.png

image.png

Solved

9 Replies

Status changed to Awaiting Railway Response Railway • 24 days ago


sam-a
EMPLOYEE

24 days ago

This is a transient build failure that resolves on its own - a subsequent deploy of the same code will succeed. We understand it's disruptive when it hits a production deploy.

You're right that healthchecks are the key protection here. With a healthcheck configured, a failed deploy never replaces the running deployment, so the production impact goes away. Adding healthchecks to your worker services will close that gap.

For retriggering a deploy after a failure, you can use the three-dot menu on the most recent successful deployment to redeploy from there. Your broader point about wanting this to not happen in the first place is well taken.


Status changed to Awaiting User Response Railway • 24 days ago


sam-a

This is a transient build failure that resolves on its own - a subsequent deploy of the same code will succeed. We understand it's disruptive when it hits a production deploy. You're right that healthchecks are the key protection here. With a healthcheck configured, a failed deploy never replaces the running deployment, so the production impact goes away. Adding healthchecks to your worker services will close that gap. For retriggering a deploy after a failure, you can use the three-dot menu on the most recent successful deployment to redeploy from there. Your broader point about wanting this to not happen in the first place is well taken.

therobotcarlson
PROOP

24 days ago

Actually, I dug into this service. It has a healthcheck. But also, this failure was during the build step, not deploy and yet it still removed the current deployment.


Status changed to Awaiting Railway Response Railway • 24 days ago


btausman
PRO

24 days ago

This issue has been occurring for me all afternoon, too. Contrary to the assertion that a redeploy will clear it up, Whether I manually redeploy via the dot menu or push an update to git that triggers the redeploy, the failed to fetch snapshot issue persists.

Reference - Railway Service ID 706cf7fe-d6b6-4931-a494-5c23f6c88145


24 days ago

We tracked this down. When a build is skipped because nothing matched your watch patterns, the source snapshot can get recorded before its upload finishes, so your next commit deploy fails looking for it rather than falling back to a full build. Retrying the deploy clears it, no new commit needed. If you hit it straight after a Redeploy instead, deploy fresh from source, since Redeploy points back at the same snapshot. Your previous deployment did keep serving throughout, it wasn't removed until your successful redeploy went live nearly three hours later.


Status changed to Awaiting User Response Railway • 24 days ago


brody

We tracked this down. When a build is skipped because nothing matched your watch patterns, the source snapshot can get recorded before its upload finishes, so your next commit deploy fails looking for it rather than falling back to a full build. Retrying the deploy clears it, no new commit needed. If you hit it straight after a Redeploy instead, deploy fresh from source, since Redeploy points back at the same snapshot. Your previous deployment did keep serving throughout, it wasn't removed until your successful redeploy went live nearly three hours later.

therobotcarlson
PROOP

24 days ago

Just in case I didn't link it above (can't tell), here's my service and the failed deploy: https://railway.com/project/7f6df885-6c07-4891-accd-09bbcb441429/service/4e7ba749-2707-4e3c-b53e-7181251bd39a?groupId=96fd4100-96e0-4d2f-9e2f-e90ebdfc5274&environmentId=aef3c71b-ccb7-43b5-a6ed-cfca5f9a50d2&id=857d0acd-69d1-4024-94b0-7c06d596cd8b

My service did not keep serving throughput as far as I can tell. I tried accessing it during this to see and landed on the Railway "your application is not responding" page. I ignored the initial alert because I (incorrectly) assumed it would keep serving since it was just a build failure. After opening it and finding it broken, I clicked around trying to get it back up until I pushed another change. That's what drove me to post, it was counter to expected behavior.

And to reiterate, there is no redeploy option when this kind of failure happens:

image.png

Attachments


Status changed to Awaiting Railway Response Railway • 24 days ago


24 days ago

Do we have consent to view your project?


Status changed to Awaiting User Response Railway • 24 days ago


brody

Do we have consent to view your project?

therobotcarlson
PROOP

24 days ago

yes


Status changed to Awaiting Railway Response Railway • 24 days ago


24 days ago

You currently have an active deployment that succeeded after the linked failed deployment. My understanding is that you are unblocked. Is that not the case?


Status changed to Awaiting User Response Railway • 24 days ago


therobotcarlson
PROOP

24 days ago

Yes, I am unblocked. Not a P0 anymore, just something that happens periodically and is unexpected when it does.


Status changed to Awaiting Railway Response Railway • 24 days ago


Status changed to Solved Railway • 24 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...