18 days ago
Hi Railway team,
I’m on a paid Railway plan. We're designing a service where exactly one worker instance may send messages to an external API at any time. If an old instance kept running, even briefly, after a new one took over, a customer could receive a message they had asked us to stop.
Our design starts every new deployment in a "sealed" mode, where it cannot send. We only unseal it after we have proof that the previous deployment is fully dead. To build that proof, we need to understand exactly what Railway guarantees. Could you please answer these questions?
- Which operation actually stops the workload? For a specific deployment ID, is deploymentStop the call that terminates the running instances, while deploymentRemove only removes it from history? Or do both terminate the workload?
- What do the status fields guarantee? When a deployment reports status REMOVED and deploymentStopped = true, is it guaranteed that every process, including child processes, in every instance of that deployment has been terminated and can no longer open or use network connections, including connections that were already open?
- Failures and partitions. Does that guarantee still hold if the host is unreachable or partitioned when the stop happens? Could an instance that Railway reported as stopped ever resume and make outbound requests after the host recovers?
- Ordering. Is there any point at which deploymentStopped = true could be reported before the instances have actually terminated?
- The right signal. If the above isn't guaranteed, is there any signal (API field, event or webhook) that does guarantee all instances of a given deployment are permanently terminated?
- Platform-initiated redeploys. When Railway itself restarts or migrates a service (for example during maintenance), can the old and new instances overlap, and for how long at most? Is there any way to be notified before or after it happens?
A written answer we can keep on file would be really appreciated, as we'll rely on it for a safety control. Thanks very much!
4 Replies
18 days ago
Use deploymentRemove, not deploymentStop. deploymentStop stops the container but changes no status, so the deployment keeps reporting SUCCESS and gives you no confirmation. deploymentRemove deletes routes, sends SIGTERM, escalates to SIGKILL after the service's draining period (default 0 seconds, set with RAILWAY_DEPLOYMENT_DRAINING_SECONDS), then destroys the container. The deployment moves to REMOVING and then REMOVED. See Manage Deployments.
REMOVED is written after teardown completes for every replica, not before it starts. Destroying the container ends every process inside it, including children, along with the connections they had open.
There is one caveat for your evidence model. If the host is unreachable, the deployment can be recorded as REMOVED because the container is believed to be absent, not because we just terminated it. We cannot guarantee in writing that a container on a partitioned host can never run again after the host recovers, so REMOVED should not be your only safety control. There is no stronger API field, event or webhook that attests permanent termination.
Railway does initiate redeploys to migrate services for host maintenance, infrastructure upgrades and security patches. For services with a volume attached, the old container is stopped before the new one starts, so the two never overlap. For services without a volume, we do not publish a maximum overlap. Your API cannot get an advance signal before a platform-initiated redeploy. Fencing on the external API's side, such as a lease or token the old worker can no longer present, is the only way to enforce exactly one sender.
Status changed to Awaiting User Response Railway • 18 days ago
18 days ago
Thank you. Could a Railway engineer please confirm these points? In particular, does deploymentStopped = true, together with REMOVED, guarantee that every replica and child process has terminated and can no longer use existing or new network connections, including during a host partition or after recovery? Or can those fields be set before termination is verified? Please also confirm whether any API signal provides that stronger guarantee, and the correct distinction between deploymentStop and deploymentRemove. We need a human-confirmed answer before relying on this as a safety control.
Status changed to Awaiting Railway Response Railway • 18 days ago
18 days ago
To confirm the distinction: deploymentStop stops the running container but does not change the deployment's status, so it keeps reporting SUCCESS and gives you no completion evidence. deploymentRemove deletes routes, sends SIGTERM, escalates to SIGKILL after the service's draining period (default 0 seconds), then removes the container. The deployment moves to REMOVING and then REMOVED, readable from the public API.
REMOVED is written after teardown completes for every replica, not before it starts. Removing the container ends every process inside it, including child processes, so the connections those processes held end with them. For your safety control, treat REMOVED as the termination signal rather than the deploymentStopped flag on its own.
The exception is a host that is unreachable. In that case the deployment can be recorded as REMOVED because the container is considered absent, not because termination was just confirmed. We cannot guarantee that a container on a partitioned host can never run again or reach the network after that host recovers. No API field, event or webhook gives a stronger guarantee of permanent termination than REMOVED.
Because of that, REMOVED should not be the only control for an exactly-one-sender requirement. Fencing on the external API's side, such as a lease or token the old worker can no longer present, is what enforces it. This also covers platform-initiated redeploys for host maintenance or migration. We send no advance signal for those, and we do not publish a maximum overlap for services without a volume.
Status changed to Awaiting User Response dizzydes90 • 18 days ago
18 days ago
Thank you for the clear and detailed response. I appreciate you confirming the limitations and recommending externally enforced fencing. This is very helpful for our design review.
Status changed to Awaiting Railway Response Railway • 18 days ago
Status changed to Awaiting User Response Railway • 18 days ago
Status changed to Solved Railway • 18 days ago