2 months ago
Hi,
I'm having an issue where Facebook's crawler (facebookexternalhit) is getting a 403 response when trying to scrape my site, but normal browser requests work completely fine.
I've already checked the following:
- No 403 errors appear in my app logs, so it's not coming from my Next.js app
- My robots.txt explicitly allows facebookexternalhit
- There's no middleware blocking requests
- curl and normal browser requests return 200 fine
It seems like the 403 is being returned at the Railway edge/proxy layer before the request even reaches my app. Is there any bot protection or IP filtering at the Railway infrastructure level that could be blocking Facebook's crawler IPs?
My domain is skolebole.dk and the specific URL failing is https://skolebole.dk/materialer/dansk/362
Thanks!
2 Replies
2 months ago
Our edge proxy does not perform any bot filtering, user-agent blocking, or WAF-style protection, so the 403 is not originating from our infrastructure. Your domain's public DNS resolves to 151.101.2.15, which is not a Railway IP and appears to belong to an intermediary CDN or proxy (Fastly range), so the 403 may be coming from whatever sits between your users and us at the DNS/network level.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
Thanks for looking into it. My DNS only has a single ALIAS record pointing directly to ay4f5vw5.up.railway.app — no CDN or proxy configured on my end. The Fastly IP must be coming from Railway's own infrastructure. Is there anything on your end that could be blocking specific user agents or IP ranges at the Fastly layer, even if it's not intentional bot filtering?
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Open Railway • about 2 months ago