Cronjob skipped occasionally
ymc182
PROOP

2 years ago

Hi, I have a cronjob running every 2 hours and it occasionally skipping a run (roughly 1 per day), no error or warning in active logs

Solved

15 Replies

brody
EMPLOYEE

2 years ago

does the task exit as soon as it's done?


ymc182
PROOP

2 years ago

no, theres no History that it has been ran


brody
EMPLOYEE

2 years ago

sorry not quite the answer I was looking for, does your code exit as soon as it's done it's task? (just in general, my question doesn't relate to railway)


ymc182
PROOP

2 years ago

yes


dwaynemac
PRO

2 years ago

by my understanding of the docs it will be skipped if it was building at the time at which it should have ran.
maybe you did a deploy at the time?


ymc182
PROOP

2 years ago

it has been ran for several runs, just occasionally skipping 1 runs in between , and it become fine later runs again for eg 2 hours:
2 2 2 2 4 2 2 2 2 4 2 2


alxc
PRO

2 years ago

Is there an ETA for a fix?


brody
EMPLOYEE

2 years ago

This thread has been escalated to the Railway team.

Status changed to Awaiting Railway Response brody over 1 year ago


alxc
PRO

2 years ago

Are there any updates? We're making a design decision today to move our cron services off Railway, or find a workaround that involves running a server 24/7 to effectively do our own cronservice. In either case, it's somewhat expensive when this feature is something that Railway advertises to support.


alxc
PRO

2 years ago

Ping @brody


brody
EMPLOYEE

2 years ago

At this time the native CRON scheduler is still known to skip jobs, so it would be best to use an in-code scheduler for the time being.


jake
EMPLOYEE

2 years ago

Hey Alex, can you link to some of the skips so I can investigate them?

We rolled out a fix for this a bit back but, looks like it's still skipping jobs rarely


jake
EMPLOYEE

2 years ago

BTW we believe we've tracked down this issue. It actually affected many other things on the platform, and recently started rearing it's head by making builds themselves slower

Cron's, with the lowest priority, actually got bounced. We've rolled out more resources and a monitor for this in the future!


alxc
PRO

2 years ago

I can confirm that we are still getting this issue. We first thought it was an issue on our side, so we did some work on the code to handle other errors, but with confidence, we can confirm from midnight Pacific today onwards this has still been an issue with skipped crons.


Status changed to Awaiting Railway Response Railway over 1 year ago


jake
EMPLOYEE

2 years ago

Hey Alex can you point an example project or spot where it didn't fire?


Status changed to Awaiting User Response Railway over 1 year ago


jake
EMPLOYEE

2 years ago

We've gone ahead and bumped up the slots here again. We don't want to say "Finally resolved", but we believe that we've got a solid handle on when this happens and can be paged in the event it does

PLEASE let us know if this happens again. The crons need to be reliable.

What causes this is particularly cron execution on "peak times". If you run a cron at midnight UTC, you're most likely to be affected by this, or on the hour, etc

We'd recommend offsetting your crons to say, the 17th minute past the hour every hour. In the future, we are likely going to do this for users automatically.


Railway
BOT

2 years ago

This thread has been marked as solved automatically due to a lack of recent activity. Please open a new thread if you require further assistance. Thank you!

Status changed to Solved Railway over 1 year ago


Loading...