Are you using a custom domain? If so, you can use Cloudflare as your DNS provider. They provide services that block and fight bots.

I'm using a custom domain but with Railway's built CDN

I'm also using Cloudflare btw but as DNS only, no proxy

Unfortunately, there isn't much you can do about it unless you proxy your traffic through Cloudflare and use WAF rules.

Railway only provides large scale DDoS protection, not WAF.

The recommendation here is to disable the built in CDN and enable the Cloudflare proxy with WAF rules?

Having both CDN will be kinda pointless

Yes, if you wish to block bot traffic.