Is there a cron job to re-deploy services?
matisidler
PROOP

2 years ago

Hey guys! So, I have some 3rd party services that are using too much RAM (no memory spikes, it slowly increases and then keeps consistent).

As I can't edit the code, it came to my mind that I can restart them every 24 hours. I was planning to use the Railway's API to achieve this, but then I thought there might be a cron job for this :)

PS: Just out of curiosity, does the V2 runtime improve the RAM consumption or is it indifferent?

45 Replies

matisidler
PROOP

2 years ago

N/A


matisidler
PROOP

2 years ago

lol I found this: https://railway.app/template/fwH-l3

brody you're the goat

will give it a try


2 years ago

hell yeah, let me know if you have any questions


matisidler
PROOP

2 years ago

man I don't know how much Railway is paying you but it's not enough


matisidler
PROOP

2 years ago

it works really well


matisidler
PROOP

2 years ago

thank you so much


2 years ago

no problem!


matisidler
PROOP

2 years ago

hey brody, is this project open source by any chance? I'd like to implement a retry system, since I'm getting these errors, which look more like a Railway's API issue

1271422392031248400


matisidler
PROOP

2 years ago

some services got redeployed, but most of them actually not


matisidler
PROOP

2 years ago

the services do exist, and I'm checking the SCHEDULE_N with the Cron Config bookmark and it's exactly the same, that's why I think it may be a Railway's issue (also because of the 503 status code lol)


2 years ago

it is, follow the source of the template service


2 years ago

do you know go?


2 years ago

looks like you where the first ever to report this issue -

but the code should do retries for a minute or however many minutes you are okay with the cron running late.


matisidler
PROOP

2 years ago

I do


2 years ago

perfect, what retry package are you thinking of?


matisidler
PROOP

2 years ago

at my job we use https://github.com/cenkalti/backoff

but happy to give a try to this one https://github.com/avast/retry-go which looks simpler


matisidler
PROOP

2 years ago

wdyt?


2 years ago

if you are more familiar with backoff then you can use that, just make sure not to hammer the api and set a max time before "giving up"


matisidler
PROOP

2 years ago

sure


matisidler
PROOP

2 years ago

what's the best way to test this?


2 years ago

something that returns 503?


matisidler
PROOP

2 years ago

I added the backoff retry, but I can't create PRs in your repo


2 years ago

oh why not?


matisidler
PROOP

2 years ago

nvm, created a PR from my fork



2 years ago

cool, I'll look over it tonight


matisidler
PROOP

2 years ago

will be using the fork for now to see how it goes


matisidler
PROOP

2 years ago

will keep you posted


2 years ago

have you tested your retries with this that returns 503


matisidler
PROOP

2 years ago

hey man! sorry I was off during the weekend

I updated the template in my project, but still having some issues:

  • The 503 issue is still there, not sure what could be causing it. I think I will try by changing the backoff params, but if it keeps returning 503 errors, the retry system is kind of useless.

  • I think the os.Exit() after a request fails could be removed. Maybe it failed to redeploy a service, but I want it to keep trying with all the other services in the list instead of restarting the container. What do you think?

Here's the project ID, just in case: b27fab0d-135f-4876-afbe-34e83cad72ac

1272513113752600600


2 years ago

The 503 issue is still there
different error handling logic is not going to fix issues with the backboard.

I think the os.Exit() after a request fails could be removed.
agree, changed, image built.


matisidler
PROOP

2 years ago

Yeah, still getting the 503 issue. I'll try changing the schedule of the cron jobs so they're not performed all at the same time


matisidler
PROOP

2 years ago

they're all being executed concurrently? could I add a delay of 30 seconds between each execution to see if this solves the issue?


2 years ago

how many schedules are you running?


matisidler
PROOP

2 years ago

around 50


2 years ago

why??


matisidler
PROOP

2 years ago

because I have multiple projects using Chatwoot and I want to restart them at 6am UTC


2 years ago

why do you need to restart Chatwoot?


matisidler
PROOP

2 years ago

the RAM usage is too high in some cases


matisidler
PROOP

2 years ago

and when I redeploy them, it decreases


matisidler
PROOP

2 years ago

I just don't want to manually redeploy them


2 years ago

yeah, then try adjusting the schedules to stagger the restarts


matisidler
PROOP

2 years ago

it worked after staggering the redeploys :)


matisidler
PROOP

2 years ago

thank you


2 years ago

then I think you where getting rate limited


Loading...