a year ago
@clientUa:Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.264 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot is sending a query to my endpoint as shown above. Normally, if it had crawled my website, this query should have been coming through Axios, right? I'm afraid that if I block Googlebot, it might block my website. Is this behavior normal? I really need an expert answer.
3 Replies
a year ago
Hello,
Completely normal for Googlebot to crawl websites on the public internet.
Completely harmless, you do not need to take any actions here.
Best,
Brody
Status changed to Awaiting User Response Railway • about 1 year ago
Status changed to Solved brody • about 1 year ago
Status changed to Awaiting Railway Response vacbid • about 1 year ago
a year ago
Thank you for your quick response. However, I believe that Googlebot crawling my endpoints is reducing the crawl budget of my main domain because these endpoints are accessed from my site, vacbid.com. Additionally, I think it is wasting my resource consumption. Is there a quick way to block this on Railway? Also, I’m asking whether this would cause any SEO issues. Thank you in advance for your answer.
a year ago
We do not offer a way to block specific requests.
For information on SEO and Google's bot, please visit their documentation.
Status changed to Awaiting User Response Railway • about 1 year ago