a month ago
Hi — our app is returning 403 to Facebook's link scraper (facebookexternalhit user-agent), but the requests never appear in our HTTP logs on railway. So the 403 is being returned at Railway's edge before reaching our Express app. This is breaking Messenger link previews for our users. How do I allowlist Facebook's scraper IPs or the facebookexternalhit UA at the edge?
Thank you!
3 Replies
a month ago
We do not have a WAF, IP-based firewall, or user-agent filtering at the edge, so the 403 is not being returned by our proxy. The most likely explanations are that Facebook's scraper is hitting a different server (check if a CDN like Cloudflare sits in front of your domain), or that your Express app itself is returning 403 through middleware that doesn't log the rejection to stdout. We also require inbound TLS with SNI, but a missing SNI would produce a connection reset, not a 403.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
We do not have a WAF, IP-based firewall, or user-agent filtering at the edge, so the 403 is not being returned by our proxy. The most likely explanations are that Facebook's scraper is hitting a different server (check if a CDN like Cloudflare sits in front of your domain), or that your Express app itself is returning 403 through middleware that doesn't log the rejection to stdout. We also require inbound TLS with SNI, but a missing SNI would produce a connection reset, not a 403.
a month ago
Thanks. We checked all three of those suspects:
No CDN — dig +short attestaclub.com resolves directly to enbey3ld.up.railway.app (69.46.46.16). No Cloudflare, no AWS, nothing in front of Railway's edge.
No Express middleware returning 403 — we reviewed app.ts; the only 403s are on /api/uploads/... (auth) and /api/* non-GET CSRF. The Facebook scraper hits /c/:slug/demande which doesn't match either.
Our HTTP Logs show only 2xx responses for that route across the entire visible window. Zero 403s ever appear. If our app were returning 403 (silently or otherwise), it would appear in the log — pino-http logs every request before the response.
So requests from Facebook's facebookexternalhit user-agent are being 403-ed before they reach our Express app, even though our logs see all other traffic to the same URL succeed with 200/304/206.
Could you check if Hikari (your edge proxy) has any IP-based rate limiting or per-IP connection limits that Facebook's scrapers might be tripping? Facebook scrapes from many IPs simultaneously and could trigger thresholds even without explicit "filtering" rules. Service: @workspace/api-server, deployment 8dd4c577.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We investigated this internally and confirmed it is a Facebook Sharing Debugger issue, not something happening at our edge. We tested with a fresh URL that Facebook had never seen, with a valid robots.txt, and our edge returned a 200 - but Facebook's debugger still reported a 403. The same behavior has been reproduced on Cloudflare, Azure, and other platforms unrelated to us. Your own logs corroborate this: your app is returning 200s for that route, and no 403 ever appears. Facebook's debugger is misreporting the status code. As a workaround, try sharing the link directly in Messenger rather than relying on the debugger's scrape - the actual in-app preview fetch often works even when the debugger claims 403.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 26 days ago