N/A

what sites are you scraping and did they give you permission?

football sites and no but their data is public

data on twitter is public but that would be a gigantic no, football sites are fine, go for it, as long as you have respectable request rates and abide by robots.txt if applicable.

ok! thanks

just to be sure, if their robots.txt doesn't allow it then Railway is against it?

User-agent: Mediapartners-Google
Disallow:

User-agent: Googlebot
Disallow:

User-agent: AdsBot-Google
Disallow:

User-agent: Googlebot-Image
Disallow:

User-agent: *
Disallow: /*?*

cc @Brody (sorry for ping, idk if the thread pops again when closed)

you gotta respect the robots.txt like any good robot would, we don't want to have to deal with takedown requests, though we will comply.

and unfortunately "it's unlikely for you to be sent a takedown request" is not an excuse

oh and you should also have an email in your UA so that web admins can email you to get put on a no crawl list

yeah I'm a bad robot so I'm guessing I should spin up this elsewhere. Could I at least host the database or the API in Railway?

yeah I don't see any issue with that