Pre-deployment check: is a scheduled domain-liveness screener (~46k hosts/6h) acceptable?
bananapeppersllc
PROOP

a month ago

Pre-deployment question per your AUP's "ask us before deploying" line. I run two sites on Railway. I'd like to add a scheduled job to the project: a registry screener that checks ~46,000 publicly listed domains for liveness roughly every 6 hours. Footprint per host: one robots.txt fetch plus at most one homepage GET, robots.txt honored, concurrency 10, identifying user-agent with a contact URL. No login pages, no per-site crawling depth, nothing beyond the homepage. Is this workload acceptable on my plan, and is there anything you'd want changed?

Solved

1 Replies

Status changed to Awaiting Railway Response Railway • about 1 month ago


a month ago

As long as you respect the rules with the robots.txt, keep a respectable request rate, and have a contact method in your user agent, you are good to go.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...