Horizontal Scaling -- Scaling controls
samoriental
PROOP

2 years ago

We're currently using Railway to host a critical node microservice that processes long-running tasks. Our architecture utilizes BullMQ for task distribution from a coordinator across multiple node workers. I would like some guidance on the ability to control how my scaling is done:

When downscaling, we need to ensure that the long tasks are not abruptly terminated, causing a data loss. Is it possible to target and gracefully kill specific replicas? This would allow us to implement a controlled downscaling process where we can ensure all tasks on a replica are completed before it's terminated.

We're considering using process.exit(0) call if it works. Would this approach work within your infra? If so, at what point in the scaling process should we trigger this - before or after making the API call to reduce the number of replicas?

Currently im planning on using the railway API to add and remove replicas but I don't know how to control how these actions are completed. Internally I can track the RAILWAYREPLICAID without an issue but I have no way to target a replica in any of the actions.

Thanks!

25 Replies

samoriental
PROOP

2 years ago

5ffdeb0b-59bc-4b91-9046-48f61f35249e


2 years ago

question, how much resources does an idle worker use?


samoriental
PROOP

2 years ago

prob like 10gb of memory because it has a couple hundred counccurrent ws connections


samoriental
PROOP

2 years ago

i guess i could kill those ws connections


2 years ago

thats a lot of memory for only a couple hundred connections…


samoriental
PROOP

2 years ago

its a lot of data lol


2 years ago

yeah but when its idle why would there be data flowing?


samoriental
PROOP

2 years ago

because accounts would be marked as inactive but still have running tasks


samoriental
PROOP

2 years ago

i could poll prisma and kill on the node


2 years ago

respectfully, this sounds like a architecture issue that you are trying to patch with replicas and a home grown autoscaler


2 years ago

ideally, you would run say 5 replicas, and each of those replicas would use a small amount of memory at idle and then each scale vertically when needed


2 years ago

imo architecting your app to run like this is going to be a far better use of your time


samoriental
PROOP

2 years ago

the issue is each nodes operations are statful but i guess keeping 10 - 20 replicas is a good idea


2 years ago

and will scale better!


samoriental
PROOP

2 years ago

i agree, thanks for the insight


samoriental
PROOP

2 years ago

also thanks for the quick support


samoriental
PROOP

2 years ago

i just have scaling to zero in my mind too much


samoriental
PROOP

2 years ago


samoriental
PROOP

2 years ago

i work a lot with stateless infra so im used to doing it this way. but i agree with what your saying and i think thats the best approach


samoriental
PROOP

2 years ago

thanks again!


2 years ago

you are on the right track, but you want your workers to be able to scale to near zero.. vertically instead


samoriental
PROOP

2 years ago

sent over some trains on the site in your bio to comp you for your time! once again i appreciate your help


2 years ago

oh wow, thank you very much!!


2 years ago

side note, you now have the pro badge


samoriental
PROOP

2 years ago

<:ok:1095784853531803700>


Loading...