a month ago
When using the RAILWAY_DEPLOYMENT_OVERLAP_SECONDS and deploying an update - what I expect to happen is that new requests get routed to the NEW deployment, while existing requests continue to be processed on the OLD deployment until the RAILWAY_DEPLOYMENT_OVERLAP_SECONDS runs out and the old server shuts down.
But, what actually happens is new requests are routed to the OLD deployment until the RAILWAY_DEPLOYMENT_OVERLAP_SECONDS runs out, only then are new requests routed properly. Since new processes will be routed to the old deployment for the duration of the overlap, then long processes will fail when the server switches to the new deployment after the set overlap.
10 Replies
a month ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
a month ago
Hi there, this is the correct behavior.
You may be looking to handle exit signals yourself and use RAILWAY_DEPLOYMENT_DRAINING_SECONDS instead? see https://docs.railway.com/guides/deployment-teardown#draining-time
Status changed to Awaiting User Response Railway • about 1 month ago
itsrems
Hi there, this is the correct behavior.You may be looking to handle exit signals yourself and use RAILWAY_DEPLOYMENT_DRAINING_SECONDS instead? see https://docs.railway.com/guides/deployment-teardown#draining-time
a month ago
ah ok thanks!
Status changed to Awaiting Railway Response Railway • about 1 month ago
itsrems
Hi there, this is the correct behavior.You may be looking to handle exit signals yourself and use RAILWAY_DEPLOYMENT_DRAINING_SECONDS instead? see https://docs.railway.com/guides/deployment-teardown#draining-time
a month ago
Ok, so using the draining env variable, I get the same issue - new requests are routed to the old server instance not the new one. This is with a 20 min draining time and a 10 second overlap time. After the 10 second overlap NEW requests get routed to the OLD server instance.
itsrems
Hi there, this is the correct behavior.You may be looking to handle exit signals yourself and use RAILWAY_DEPLOYMENT_DRAINING_SECONDS instead? see https://docs.railway.com/guides/deployment-teardown#draining-time
a month ago
There seems to be som conflicting information about this. Here: https://station.railway.com/questions/deployment-cut-over-process-a303aad8 an employee says that new requests are routed to the NEW server during overlap.
Status changed to Awaiting User Response Railway • about 1 month ago
brody
Can you please use a health check, and then report back?
a month ago
Alright I setup a health check and still the same issue.
I do see the POST request logging on the newest instance logs (see attached), but the actual processing is done on the old instance.
Only after the draining time is complete and the old instance completely shutdown, do requests get routed to the new one.
Attachments
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Because your workers are still accepting jobs in your previous deployment.
You will need to handle the SIGTERM signal and then have the workers stop accepting any more jobs. This will allow the newly deployed workers to pick up the jobs while allowing the jobs that are already running to complete.
Status changed to Awaiting User Response Railway • about 1 month ago
brody
Because your workers are still accepting jobs in your previous deployment.You will need to handle the SIGTERM signal and then have the workers stop accepting any more jobs. This will allow the newly deployed workers to pick up the jobs while allowing the jobs that are already running to complete.
a month ago
Ah ok thank you! Had this setup already but I found a bug on closer inspection - working as expected now, thank you!!
Status changed to Awaiting Railway Response Railway • about 1 month ago
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved brody • about 1 month ago