2 years ago
We have a memory leak that slowly builds over time depending on how active it is. It seems to be an issue caused by python itself so we are stuck with restarts. Plan was to have a cron job that checks the memory usage and restart if it exceeds a certain level. Is this a good idea or is something else better?
9 Replies
2 years ago
I would use this template - https://railway.app/template/fwH-l3
Set it to restart your service every day when you have your lowest amount of traffic, like 4 am or something.
2 years ago
yeah the one issue is that we may be forced to restart during high activity. We run heavy compute tasks which can quickly spike the ram and thats why we have to restart them
2 years ago
Then you would have to write a similar service to the one above that would monitor your service's memory and restart it if it reaches a certain level.
Ideally, you fix the memory leak though lol.
2 years ago
yeah its a big issue that i suspect won't be resolved any time soon https://github.com/python/cpython/issues/109534
2 years ago
you created your own service right based on this? https://blog.railway.app/p/cron-jobs.
Looks like we need to build the cron service separate from the native so we can restart and not trigger redeploys based on the native cron service
2 years ago
@brody which api does https://railway.app/template/fwH-l3 use for the restarts? railway documentation only shows deploymentRestart which im guessing restarts all services in the deployment? I am trying to restart just one specific service. they have serviceInstanceRedeploy but nothing for restarting
2 years ago
As the name may suggest deploymentRestart is for restarting a deployment, and that is what my code uses to restart a deployment for a given service.
2 years ago
okay thanks!