N/A

twitter scraper?

no, the football thing

is your bot a good bot

if it does respect robots.txt? no, but clearly i wont ddos the site with requests

would the robots.txt rules disallow your bot?

it uses a user-agent: * to disallow the paths i want to go

hmmmm

and btw the website i want to scrape gets its data from a bigger company (by scraping)

so they're scraping anyway

is there really no API for the data you want?

not from the official company

its EA btw

what's the max rps your bot would do

it depends, for scraping i can reduce it to 10 or 30 rps (it only scrapes in intervals bigger than one hour so i can reduce it no problem). i also download some images.

there's a single specific endpoint that needs to be in realtime when my customer requests it but only premium users would have access to it

and if im correct, most endpoints im getting the data from seems to be cached in Cloudflare anyway

should be no harm then

but yes we will not hesitate to take it down if we receive a report

ok and another question, would my project get taken down or my whole account?

whole team

we’re walking the line of a big big if here

if you’re not abusing the site, then they won’t issue a takedown request

therefore, railway won’t take down your app

If your app doesn’t respect the robots.txt, you’re taking a risk

but that risk isn’t huge

30 rps is a lot though, so up to you to implement mitigations

Is there a reason why you need updates at that frequency?

could you update at a lower frequency/get your data from a different source?

if you know they’re getting their data from EA at a high frequency, why not do the same?

30 rps is just the max peak it might get, pretty sure that most of the time it'll be below that

there isn't any other source available that gets all the data I need, and yes for now the interval is about every 12 hours but I might decrease that if needed by any customer

it needs auth and overall it's a mess to deal with it but in the future maybe

and just being sure: yes I'm taking into account any request I made to the server to not overwhelm it

so final answer, run it on railway

!s