2 months ago
What is recommended best practice to not lose running work or active jobs when a service redeploys?
Looks like Railway has some kind of builtin shutdown timer which I suppose would allow custom code to stop / resume jobs but that will be a lot of work.
Are there any better options?
I make changes to my service, dashboard updates all the time and waiting till overnight or something isn't an option.
2 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
You can use the Deployment Teardown feature that let you run custom code on SIGTERM event on previous deployment before it being forcefully stopped, You can customize the Overlap time yourself by configure it through the Service Settings or Service Variable RAILWAY_DEPLOYMENT_OVERLAP_SECONDS
a month ago
Railway’s draining setting is probably what you want here. On redeploy the old instance gets SIGTERM, then Railway waits for RAILWAY_DEPLOYMENT_DRAINING_SECONDS before killing it. Handle SIGTERM by stopping new jobs and letting the current one finish.
RAILWAY_DEPLOYMENT_OVERLAP_SECONDS is different — it just keeps old and new deploys running together for a while. If jobs can run longer than your drain window, then you’ll still need a persistent queue/resume strategy