Cron job stuck running...
jwaltz
PROOP

a year ago

For some reason my cron job is stuck. New executions haven't run for the last 3 hours (5 missed invocations).

I have attached screenshots of the cron runs tab and the deployment logs.

The cron job itself is super simple: it runs every 30mins and decides based on application code if it should make a fetch request or not at the appropriate time.

29 Replies

jwaltz
PROOP

a year ago

7440ea60-c734-47cc-ba5d-e8cabf480488


jwaltz
PROOP

a year ago

Going back through the logs I just noticed that it did this previously, from 9/17 3:30pm till today 9/26 5:00am which is nearly 9 days of missed executions. Thankfully this isn't in production yet but it will go live in a few months and I was hoping for it to be quite reliable.


What happens when you attempt to revert it?


To the old logic


There should be a feature flag within the service that you can enable that will restore the old cron logic


jwaltz
PROOP

a year ago

Something weird was happening where the interface was prompting me to manually approve deployments, all of the deployments that hung between 9/17 and 9/26 because they came from an outside contributor (but they didn't, just me)


jwaltz
PROOP

a year ago

I recreated the service and things seem to be running smoothly right now


jwaltz
PROOP

a year ago

The issue I am now facing is that certain executions of the cron job seem to take roughly 30 minutes. I have attached 3 examples (all occurred in the last 24hrs) where the cron job took a long time. Almost all executions are under 10s, most 5s or less.

1289697761096241200
1289697761427587000
1289697761775849500


a year ago

Sorry how is this 30 minutes to run?


a year ago

These look like multiple different runs


jwaltz
PROOP

a year ago

I don't know why, or why the logs look that way. They do look like multiple runs to me as well, but that's the logs from a single execution


a year ago

Oh, crons now run multiple times per deployment instead of creating new deployments


a year ago

You'll want to check the "Cron runs" tab


a year ago

And that will show you the logs for each "run"


jwaltz
PROOP

a year ago

That's where I got those logs from, the cron runs tab.


a year ago

Can you link me your project sorry?


a year ago

Cause, your logs look correct. I'm assuming you're running this every 30 minutes


a year ago

Oh ssorry found it aboev


a year ago

1289742028715982800


a year ago

Looks correct?


a year ago

Oh I see what you mean. That time is definitely incorrect…

But it looks like it's spanning multiple executions, and each run is running correctly


a year ago

But ye this is cosmetic BTW


jwaltz
PROOP

a year ago

That's what I figured. Something about the presentation of runs and their logs are getting stuck together, since the duration of execution seems to be the 30min interval plus another execution


jwaltz
PROOP

a year ago

Thank you for investigating


a year ago

Yep. I think this is the bug we saw where, rarely, an event just won't come through for the "end"

We'll fix that


a year ago

Heads up I'm 99% certain I fixed this issue in the last 24h. I see your last occurance was Oct 3rd at 1:30am PST

PLEASE let us know if you see this again!


jwaltz
PROOP

a year ago

Great work, thanks Cooper!


a year ago

Hey don't thank me yet! We gotta validate it over a while

It looked like it was getting that 30 minute "time" once per day, so imma check over the next few days


a year ago

But again plz do let me know if you see anything wonky


Loading...