a month ago
In our production environment, the web and worker services run through npm as PID 1. We need to pause public ingress, drain accepted requests and jobs, then stop all old instances before deployment migrations run.
What supported procedure stops these current deployments without redeploying? Does scale-to-zero drain accepted requests, honor the saved 360-second grace period, and prevent old instances restarting during migration?
7 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This is the kind of problem the Railway community is well placed to help with, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public and the thread closes.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
Status changed to Solved 504naldo • about 1 month ago
a month ago
I accidentally marked this thread as solved. The issue is still unresolved. Could you please reopen it so the community can help?
Status changed to Open Railway • about 1 month ago
a month ago
I think you need to configure deployment teardown in this case: https://docs.railway.com/deployments/deployment-teardown. But you need to run your web and worker using node directly, instead of a package manager (like npm in this case), so it can handle the SIGTERM properly: https://docs.railway.com/deployments/troubleshooting/nodejs-sigterm-handling. It's also recommended that you run migrations in a pre-start command.
And if you want to remove a deployment manually, you can do so through the deployment tab in your service settings, from the 3 dots, select the option Remove.
a month ago
Dmohon untuk berhati-hati
darseen
I think you need to configure deployment teardown in this case: https://docs.railway.com/deployments/deployment-teardown. But you need to run your web and worker using node directly, instead of a package manager (like npm in this case), so it can handle the SIGTERM properly: https://docs.railway.com/deployments/troubleshooting/nodejs-sigterm-handling. It's also recommended that you run migrations in a pre-start command. And if you want to remove a deployment manually, you can do so through the deployment tab in your service settings, from the 3 dots, select the option **Remove**.
a month ago
Thanks. That helps with future deployments, but our current web and worker services already run through npm as PID 1. We need to pause incoming traffic and new jobs, drain accepted work, and stop all existing instances before database migrations run. What supported procedure does that without triggering another deployment? Does it honor the shutdown grace period, and how can we verify all old instances have stopped?
504naldo
Thanks. That helps with future deployments, but our current web and worker services already run through npm as PID 1. We need to pause incoming traffic and new jobs, drain accepted work, and stop all existing instances before database migrations run. What supported procedure does that without triggering another deployment? Does it honor the shutdown grace period, and how can we verify all old instances have stopped?
a month ago
You can delete the public URL for the web, and worker temporarily. This will effectively stop all incoming public traffic, and drain all accepted work. You can then remove the old deployments, and trigger new ones with the database migrations.
Note: if your services use the generated railway URL, you need to save it before deleting, so you edit it back like it was (because it might generate a new URL after deleting the old one).
a month ago
AI-assisted answer; checked against the linked Railway documentation, but not tested against your production deployment.
Railway now documents a way to remove replicas without a redeploy: railway scale. Setting a region to zero removes its replicas; the command applies the replica configuration without a separate redeploy. Account for every configured region of both services, not just one region. The scaling guide says removed replicas are drained gracefully.
That is not, by itself, a guarantee that your accepted requests/jobs finish:
- Railway's Node SIGTERM guidance specifically identifies npm as a reason the application may never receive SIGTERM. Scaling cannot retrofit a shutdown handler or change the process tree of the already-running app.
- The 360-second draining setting is documented in the replacement-deployment lifecycle. I could not verify from these docs whether scale-down or explicit stop uses that saved value for your current instances. Railway needs to confirm that detail before you rely on it.
- Removing a public domain is not evidence that queue consumers have stopped fetching jobs, private callers have stopped, or accepted work has finished.
For this maintenance window, separate application drain from infrastructure stop:
- Hold deployment triggers and other actors that could bring services back while migrations run. Inventory the current deployment IDs and replicas across all regions.
- Use an existing maintenance/drain control to reject new requests, and the queue's supported mechanism to pause job acquisition. Wait for application in-flight counters and active jobs to reach zero on every instance. If the current application has no usable drain mechanism and its Node process cannot receive shutdown signals, I cannot offer a guaranteed lossless procedure for those instances without changing the application or accepting interruption/retry.
- Once drained, set the replica count to zero in every configured region of each service. Alternatively, the public API documents
deploymentStop(id: ...)for stopping a running deployment; it is distinct from redeploy and from removing history. Do not treat an acknowledged request as proof that termination has completed. - Before migrations, verify zero running old replicas in Railway, completed application/worker shutdown, and no old database writers. Quiet logs or an unreachable public URL alone are insufficient. Keep the deployment hold until the migration and controlled restart finish.
A pre-deploy migration alone is not a stop-all-old-instances barrier: pre-deploy runs before the new app deploys, while replacement teardown occurs after the new deployment becomes active.
Railway team: can you confirm the timeout used by scale-to-zero/deploymentStop for already-running instances, and the authoritative completion signal covering every replica? That is the platform-specific guarantee still missing here.