Hey everyone!!

I'm prototyping a data pipeline app: a Python backend that scrapes Instagram posts, hashtags, and comments using Apify actors. It then runs sentiment analysis on the data with LLMs, saves everything to a SQLite DB, and I've got a separate Svelte frontend to pull and display the results.

The goal is to automate this whole flow to run once a day, say at midnight UTC. Right now, I don't have a solid benchmark on how long it'll take—could be a quick 30 mins if things are light, or stretch to 2-4 hours if the scrapes hit rate limits or the analysis gets heavy with more data.

From the docs, cron jobs look promising: no hard timeout, meant for tasks that finish and exit cleanly, skips overlaps (not an issue daily), and 5-min min interval. But I wanted to check with folks who've done similar: Would a potentially long job like this run reliably without getting killed, surprise bills piling up, or just getting skipped? Any gotchas with resource handling over extended periods?

If crons aren't the move for something this variable, what other Railway features could I lean on? Like job queues, background workers, or hooking into external schedulers via webhooks? Open to any tips to make this workflow play nice here.

Happy to share more info on the setup if that helps. Thanks a ton!!