Cron Runs on Old Deployment

nickcherryPRO

8 months ago

I have a cron service linked to a Github repository. When a new commit lands on main, the service seems to automatically deploy the latest code without issue, judging by the feedback on the "Deployments" tab. The "Cron Runs" tab also confirms that the "Current Deployment" is the most recent commit, and its deployment status is "Active". However, when i press "Run now" to immediately start the cron job (for testing/debugging), it appears to be using an older commit, based on the "Recent Executions" list item that appears. The only way I have figured out to run the latest code is to delete and recreate the service entirely. The behavior I would expect is that the cron job will always run on the latest deployment. Am I doing something wrong / misunderstanding the expected behavior, or is this a bug?

Awaiting User Response

8 Replies

8 months ago

Hi Nick,

Thank you for your report. I've escalated it to our product team for investigation.

Regards,

Christian


Status changed to Awaiting User Response railway[bot] 8 months ago


nickcherryPRO

8 months ago

Thanks, Christian


Status changed to Awaiting Railway Response railway[bot] 8 months ago


8 months ago

Unfortunately I wasn't able to reproduce this.

What I did to test -

- Wrote code to print the GitHub commit message via RAILWAY_GIT_COMMIT_MESSAGE

- Committed that code to GitHub with an init message

- Deployed it to Railway and ran it on a 5 minute CRON schedule

- Pushed an empty commit with an `empty commit` message

- Further CRON schedule now ran the code from this new commit and printed the correct message

- Clicked Run Now, the code that ran immediately used the latest commit and printed the correct message of empty commit

Please let me know if my testing is flawed or is not representing what you have done.


Status changed to Awaiting User Response railway[bot] 8 months ago


nickcherryPRO

8 months ago

Thanks for looking into it, Brody. The issue is still very much alive and well for me. I'm not 100% sure, but I believe it may specifically occur after there is a bad deployment. For example:

1. Configure and deploy a service that works fine
2. Later deploy a bad commit (e.g. one that fails on bun install)
3. Commit fixes to resolve the deployment
4. Cron continues to try to run broken deployment

You can see it breaking currently with this service:

https://railway.app/project/0b4426fc-05c1-4da1-9622-6af85e7c1298/service/c241c0ba-5540-4611-935e-326f8060eb32/schedule

I've also attached screenshots showing what I see in the UI. The re-add patch commit is the one that broke the deployments (on a side note, it seems Bun's patch does not work in a Docker/Railway environment), while the clean up models commit is working. The UI is a bit confusing here, because it says the recent execution is clean up models (which is correct), however the current deployment is re-add patch (incorrect). I'm also receiving email notifications saying the build has failed, and looking at the database, it seems that is the case.

Attachments


Status changed to Awaiting Railway Response railway[bot] 8 months ago


nickcherryPRO

8 months ago

Just got another failure: https://railway.app/project/0b4426fc-05c1-4da1-9622-6af85e7c1298/service/c241c0ba-5540-4611-935e-326f8060eb32?id=096fb5d8-9644-4241-9dc5-0b9cd5a19452


8 months ago

With this new information, I wasn't able to reproduce, but I noticed you enabled app sleeping? i'd be curious to know why you have done that.

Either way, this misconfiguration is likely what has your cron in a broken state, so please go ahead and duplicate the service, this can be done via right clicking on the service and choosing duplicate.


Status changed to Awaiting User Response railway[bot] 8 months ago


nickcherryPRO

8 months ago

I'm pretty sure I tried removing the sleep configuration, which had no effect at resolving the issue. I'll leave it off in the duplicated services and see if the problem persists.

I haven't seen documentation that makes any reference of enabling sleeping with a cron task one way or another. Assuming that the purpose of the sleep behavior is to conserve resources, background jobs intuitively seem like the ideal use case for it – the overwhelming majority of the time, the services are doing nothing, and unlike a web server, you typically aren't going to care about cold start costs. I have no idea how Railway is architected, but if the jobs are being started by a service that's always on (i.e. not relying on cron on the server in question), I suspect it would be trivial to "wake up" the sleeping service.


Status changed to Awaiting Railway Response railway[bot] 8 months ago


8 months ago

CRON jobs on Railway do not keep your service running 24/7.

Whenever the job is not running, nothing is running.

Your code is only running and you are only being charged when the task is running.


Status changed to Awaiting User Response railway[bot] 8 months ago