Facebook crawler (facebookexternalhit) getting 403'd at the edge, never reaches my service
Anonymous
FREEOP

17 days ago

I have posted this threid and I post it again for it still dosent works and it have been closed, the last massage from the customer service said that the problem is in their servers and they will work on it but it still doesnt work

Hi,

I'm running into an issue where Facebook's link-preview crawler (facebookexternalhit) can't fetch my site, which breaks link previews when shared on Facebook/Instagram.

Project: Frontend service, domain www.thecompassmedia.net

What's happening:

When I run my URL through Facebook's Sharing Debugger (https://developers.facebook.com/tools/debug/), it gets a 403 response.

Normal browser traffic to the same domain works fine (200s in my HTTP Logs for /, /robots.txt, /sitemap.xml, assets, etc.).

My robots.txt does not block any user-agents, and there's no firewall/bot-blocking logic in my own application code — I've verified this directly in the source.

I checked my Railway HTTP Logs for the Frontend service while triggering "Scrape Again" in the Facebook Debugger, and no request from facebookexternalhit appears in the logs at all — not even a 403 entry. This tells me the request is being blocked before it reaches my service, somewhere at Railway's edge.

What I'm asking:

Could you check whether there's bot/crawler filtering happening on Railway's edge/proxy layer for my custom domain (www.thecompassmedia.net), and allowlist Meta's crawler (facebookexternalhit / Facebot user-agents, and/or Meta's published crawler IP ranges) so Facebook can fetch and preview my pages?

Happy to provide the project ID, request IDs, or timestamps of my scrape attempts if that helps you trace it on your end.

Thanks for your help!

Solved$10 Bounty

2 Replies

Railway
BOT

17 days ago

Your domain and certificate are healthy. The symptoms you describe (browser traffic passes, non-browser crawler traffic gets 403, no request in your service logs) match Under Attack Mode being active on your Frontend service. When enabled, it blocks all non-browser traffic at the edge before it reaches your app, which would explain why Facebook's crawler is rejected and never appears in your logs. Check your Frontend service's Settings, then the Edge section, and if Under Attack Mode is shown as active, deactivate it.


Status changed to Awaiting User Response Railway 17 days ago


Railway

Your domain and certificate are healthy. The symptoms you describe (browser traffic passes, non-browser crawler traffic gets 403, no request in your service logs) match [Under Attack Mode](https://docs.railway.com/networking/waf) being active on your Frontend service. When enabled, it blocks all non-browser traffic at the edge before it reaches your app, which would explain why Facebook's crawler is rejected and never appears in your logs. Check your Frontend service's Settings, then the Edge section, and if Under Attack Mode is shown as active, deactivate it.

Anonymous
FREEOP

17 days ago

yes we have checked that earlier and it is not activated


Status changed to Awaiting Railway Response Railway 17 days ago


Railway
BOT

15 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 15 days ago


Status changed to Solved Anonymous 13 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...