6 months ago
@clientUa:Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.264 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot is sending a query to my endpoint as shown above. Normally, if it had crawled my website, this query should have been coming through Axios, right? I'm afraid that if I block Googlebot, it might block my website. Is this behavior normal? I really need an expert answer.
3 Replies
6 months ago
Hello,
Completely normal for Googlebot to crawl websites on the public internet.
Completely harmless, you do not need to take any actions here.
Best,
Brody
Status changed to Awaiting User Response railway[bot] • 6 months ago
Status changed to Solved brody • 6 months ago
Status changed to Awaiting Railway Response vacbid • 6 months ago
6 months ago
Thank you for your quick response. However, I believe that Googlebot crawling my endpoints is reducing the crawl budget of my main domain because these endpoints are accessed from my site, vacbid.com. Additionally, I think it is wasting my resource consumption. Is there a quick way to block this on Railway? Also, I’m asking whether this would cause any SEO issues. Thank you in advance for your answer.
6 months ago
We do not offer a way to block specific requests.
For information on SEO and Google's bot, please visit their documentation.
Status changed to Awaiting User Response railway[bot] • 6 months ago