3 days ago
Hi,
I'm running into an issue where Facebook's link-preview crawler (facebookexternalhit) can't fetch my site, which breaks link previews when shared on Facebook/Instagram.
Project: Frontend service, domain www.thecompassmedia.net
What's happening:
When I run my URL through Facebook's Sharing Debugger (https://developers.facebook.com/tools/debug/), it gets a 403 response.
Normal browser traffic to the same domain works fine (200s in my HTTP Logs for /, /robots.txt, /sitemap.xml, assets, etc.).
My robots.txt does not block any user-agents, and there's no firewall/bot-blocking logic in my own application code — I've verified this directly in the source.
I checked my Railway HTTP Logs for the Frontend service while triggering "Scrape Again" in the Facebook Debugger, and no request from facebookexternalhit appears in the logs at all — not even a 403 entry. This tells me the request is being blocked before it reaches my service, somewhere at Railway's edge.
What I'm asking:
Could you check whether there's bot/crawler filtering happening on Railway's edge/proxy layer for my custom domain (www.thecompassmedia.net), and allowlist Meta's crawler (facebookexternalhit / Facebot user-agents, and/or Meta's published crawler IP ranges) so Facebook can fetch and preview my pages?
Happy to provide the project ID, request IDs, or timestamps of my scrape attempts if that helps you trace it on your end.
Thanks for your help!
5 Replies
3 days ago
We don't have a general bot-filtering layer at the edge, but our Under Attack Mode (in the service's Settings > Edge section) blocks all non-browser traffic when active, which would produce exactly the symptoms you're seeing - a 403 for crawlers with no entry in your service logs. Please check whether Under Attack Mode is enabled on your Frontend service and deactivate it if so; Facebook's crawler should then be able to reach your site normally.
Status changed to Awaiting User Response Railway • 3 days ago
Railway
We don't have a general bot-filtering layer at the edge, but our [Under Attack Mode](https://docs.railway.com/networking/waf) (in the service's Settings > Edge section) blocks all non-browser traffic when active, which would produce exactly the symptoms you're seeing - a 403 for crawlers with no entry in your service logs. Please check whether Under Attack Mode is enabled on your Frontend service and deactivate it if so; Facebook's crawler should then be able to reach your site normally.
3 days ago
I have checked that but it was deactive so it should be somthing else that cose this problem
Status changed to Awaiting Railway Response Railway • 3 days ago
Railway
We don't have a general bot-filtering layer at the edge, but our [Under Attack Mode](https://docs.railway.com/networking/waf) (in the service's Settings > Edge section) blocks all non-browser traffic when active, which would produce exactly the symptoms you're seeing - a 403 for crawlers with no entry in your service logs. Please check whether Under Attack Mode is enabled on your Frontend service and deactivate it if so; Facebook's crawler should then be able to reach your site normally.
3 days ago
I've confirmed Under Attack Mode is not active on my Frontend service, but I'm still getting 403 responses for facebookexternalhit in Facebook's Sharing Debugger, with no requests appearing in my service logs.
Since the crawler is being blocked at the edge (not by my application), could you:
Check if there are any other edge filters, WAF rules, or security policies blocking Meta's crawlers on my domain (www.thecompassmedia.net)?
Verify the routing/proxy configuration for my custom domain to ensure it's not filtering non-browser traffic elsewhere?
Provide the exact steps to debug what's blocking these requests at the edge?
Project ID: ab73d069-6d2b-43ee-ab31-3d8ec90d50c8
3 days ago
This is a known block on our side. Meta's crawler is being stopped at our edge before it reaches your service, and there is no change on your end that resolves it. Our team is tracking the edge-level block affecting Meta's crawler.
Status changed to Awaiting User Response Railway • 3 days ago
noahd
This is a known block on our side. Meta's crawler is being stopped at our edge before it reaches your service, and there is no change on your end that resolves it. Our team is tracking the edge-level block affecting Meta's crawler.
2 days ago
so I just need to wait and you will solve the problem soon ?
Status changed to Awaiting Railway Response Railway • 2 days ago
Status changed to Closed brody • 1 day ago