a year ago
We're currently using Railway to host a critical node microservice that processes long-running tasks. Our architecture utilizes BullMQ for task distribution from a coordinator across multiple node workers. I would like some guidance on the ability to control how my scaling is done:
When downscaling, we need to ensure that the long tasks are not abruptly terminated, causing a data loss. Is it possible to target and gracefully kill specific replicas? This would allow us to implement a controlled downscaling process where we can ensure all tasks on a replica are completed before it's terminated.
We're considering using process.exit(0) call if it works. Would this approach work within your infra? If so, at what point in the scaling process should we trigger this - before or after making the API call to reduce the number of replicas?
Currently im planning on using the railway API to add and remove replicas but I don't know how to control how these actions are completed. Internally I can track the RAILWAYREPLICAID without an issue but I have no way to target a replica in any of the actions.
Thanks!
25 Replies
a year ago
question, how much resources does an idle worker use?
prob like 10gb of memory because it has a couple hundred counccurrent ws connections
a year ago
thats a lot of memory for only a couple hundred connections…
a year ago
yeah but when its idle why would there be data flowing?
a year ago
respectfully, this sounds like a architecture issue that you are trying to patch with replicas and a home grown autoscaler
a year ago
ideally, you would run say 5 replicas, and each of those replicas would use a small amount of memory at idle and then each scale vertically when needed
a year ago
imo architecting your app to run like this is going to be a far better use of your time
the issue is each nodes operations are statful but i guess keeping 10 - 20 replicas is a good idea
a year ago
and will scale better!
i work a lot with stateless infra so im used to doing it this way. but i agree with what your saying and i think thats the best approach
a year ago
you are on the right track, but you want your workers to be able to scale to near zero.. vertically instead
sent over some trains on the site in your bio to comp you for your time! once again i appreciate your help
a year ago
oh wow, thank you very much!!
a year ago
side note, you now have the pro badge