Loading...

Cron Job Execution Limits for Production Scraping Use Case

sbdeals

PROOP

2 months ago

Hello,
I'm evaluating Railway for a production scraping platform that needs to run scheduled data collection jobs daily. Before committing to Railway, I need clarification on your cron job execution parameters:

Here's my use case:

Daily scheduled scraping jobs with 100+ runs
Estimated execution time: 15-30 minutes per job currently, potentially 60-90 minutes as we scale
Python-based scrapers that POST data to external database (Supabase)
Jobs need to run reliably without hitting timeout limits

Questions:

What is the hard limit for cron job execution duration? The documentation mentions "short-running tasks" but doesn't specify exact timeouts.
Are there memory constraints for scheduled cron jobs?
What happens if a cron job exceeds the time limit? (Hard kill, graceful shutdown with error notification, etc.)
Are there any rate limiting policies for scheduled executions that could impact daily jobs?
If we need to run multiple parallel scraping jobs (different stores simultaneously), what are the concurrency limits?

I'm choosing between Railway's managed solution and a traditional VPS, and your clarification on these limits is crucial for my decision. Thank you for your help!

Solved

2 Replies

Railway

BOT

2 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!

brody

EMPLOYEE

2 months ago

Hello,

I'll try to answer your questions to the best of my ability.

There isn't any time limits, you only need to make sure the job completes by the time the next job starts.
None specific to cron jobs; your standard CPU and memory limits allowed by your plan apply to cron jobs the same as they do to any other service.
There is no limit whatsoever, it could run indefinitely; your application is responsible for stopping itself. However, if your job runs over into the time the next job would start, the next job would be skipped.
None. The only thing to note would be the minimum of a 5-minute interval between runs.
We have no intrinsic parallelization limits; the bottleneck would be your code's ability to vertically scale to the resource limits of your plan.

Given these answers, it is clear that CRON jobs are more than capable of handling your current use case, and any scaling you will do in the future.

Best,
Brody

Status changed to Awaiting User Response Railway • about 2 months ago

Status changed to Solved sbdeals • about 2 months ago