a month ago
Hello!
We are working with multiple services within Railway for different components of our app. Currently we have 4 services, which communicate with each other via HTTP or via a queue.
If we change HTTP contracts or message queue contracts, this can result in downtime if one of the services deploys first while the dependency is still building or deploying.
What we want ideally is for all containers to build, and be in a state where they switch over to the new commit together.
Yes, there are edge cases that can occur here, but they're blips that can be dealt with a simple refresh or retry instead of having customer stuck with a non-responsive endpoint for 4-5 minutes.
I don't know if Railway already provides this mechanism or maybe we're missing something simple, but I couldn't see it in the UI, docs or the forum. Would appreciate some help on this, thanks!
4 Replies
a month ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
a month ago
Hello,
I believe using Reference Variables should help with this. It will allow services to deploy after a service they depend on has been deployed. You can see more about this here - https://docs.railway.com/guides/deployment-actions#deployment-dependencies---startup-ordering
Let me know if that works for you!
Best,
a month ago
Thanks for sharing. This would work well for some cases, but quoting from the section: "Services that have circular dependencies will simply ignore them and deploy as normal."
Our services often talk back and forth with each other over HTTP, and sometimes over a message queue, depending upon the specific operation. In such cases, they both refer to each other, and this will result in issues afaik.
a month ago
It sounds like in this case the services would depend on at least the message queue being online first. That said it's not possible to set up a specific start order of services with Railway. You may however be interested in taking a look at the response to a similar thread here where it's suggested to use a script that first checks the availability of a given service and if it's available to then start itself. You could likely modify the script as needed to have it check for several services before it starts. With circular dependencies it'll be rather tough though since there isn't functionality to start all the new containers at once for every new deployment.
After thinking about it for another moment, an idea that popped into my head.
You could write a script that is included in each of the containers and is run at start.
The script would use something like
sleep 300(5 minutes) to ensure all new containers have had enough time to have been built and deployed. You could adjust the sleep time as you see fit.Once the 5 minutes is up the script will start checking to see if the other 3 containers are up and their services are available.
If they are then that containers will break out of the checking loop and start its service.
The checking part of the script could run every few seconds which should make the new deployment come up quickly and closely with the other new containers.
While I understand this isn't your ideal situation I hope it helps to put you closer to what you're looking to achieve!