10 days ago
Facebook's scraper getting 403 from Railway-hosted Next.js app — server returns 200
Platform: Railway (Next.js 14, App Router)
Issue: Facebook's Open Graph scraper (facebookexternalhit/1.1) reports a 403 when hitting our homepage, but our app returns HTTP 200 to that user agent.
What we've confirmed:
- Tested with
Invoke-WebRequestspoofing Facebook's user agent directly against the live Railway deployment:
Invoke-WebRequest -Uri "https://dogwalkingtorbay.co.uk" -Method HEAD -Headers @{"User-Agent"="facebookexternalhit/1.1"} → Returns HTTP 200
- No middleware blocking Facebook — Next.js middleware matcher is scoped to
/admin/:path*only robots.txtexplicitly allowsfacebookexternalhit- No IP blocking or rate limiting configured anywhere in our code
- Facebook's debugger at developers.facebook.com/tools/debug consistently returns 403
- Other scrapers (WhatsApp, LinkedIn, Twitter) work fine
Question: Is Railway doing any infrastructure-level filtering or blocking of Facebook's datacenter IP ranges before requests reach the application? The 403 appears to originate after the response leaves our app but before it reaches Facebook — which points to something at the network/proxy layer.
Happy to share the specific Facebook DC IPs it's hitting from if that helps narrow it down.
1 Replies
10 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 10 days ago
Status changed to Awaiting Railway Response Railway • 5 days ago
Status changed to Awaiting User Response Railway • 5 days ago
5 days ago
I would not start by assuming Railway is blocking Facebook here. The redirect chain is suspicious.
From an external check with Facebook's UA, the apex domain first responds from a non-Railway server:
curl -IL -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' https://dogwalkingtorbay.co.uk
Observed shape:
HTTP/2 301
server: LiteSpeed
x-redirect-by: WordPress
location: https://www.dogwalkingtorbay.co.uk/
HTTP/2 200
server: railway-edge
x-railway-edge: railway/us-east4-eqdc4a
So https://dogwalkingtorbay.co.uk is not reaching Railway first. It is hitting an old/other LiteSpeed + WordPress origin that redirects to www, then www reaches Railway. If Facebook's datacenter IPs are getting 403, the most likely source is that first apex-domain hop or cached Meta state, not Railway's app/proxy.
Things I would do:
- Test the exact URL submitted to Facebook. If you submit the apex URL, Meta hits the LiteSpeed/WordPress hop before Railway. Submit/scrape
https://www.dogwalkingtorbay.co.uk/directly. - Move the apex domain onto Railway too, or make the apex redirect happen from your Next/Railway app instead of the old LiteSpeed/WordPress host.
- In DNS, verify
dogwalkingtorbay.co.ukis not still pointed at the old host whilewwwpoints to Railway. Both should be intentionally configured. - Keep
og:urlcanonical ashttps://www.dogwalkingtorbay.co.ukifwwwis the real canonical URL. Your current page already serves that value from Railway. - In Facebook Sharing Debugger, use "Scrape Again" after the DNS/redirect change because Meta can cache old HTTP failures.
- Test with GET, not only HEAD. Facebook generally fetches the HTML with GET, then fetches the OG image. The OG image currently returns 200 from Railway for the Facebook UA.
Useful commands:
curl -IL -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' https://dogwalkingtorbay.co.uk
curl -L -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' https://dogwalkingtorbay.co.uk -o /dev/null -w 'code=%{http_code} final=%{url_effective}\n'
curl -IL -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' https://www.dogwalkingtorbay.co.uk/og-image.jpg
If Facebook still returns 403 after the apex is removed from the path, then collect the Railway request IDs for a Facebook scrape attempt. But right now the visible chain shows an upstream non-Railway hop before the Railway app.
Status changed to Awaiting Railway Response Railway • 5 days ago
Status changed to Awaiting User Response Railway • 5 days ago