Fastly bot protection blocking Facebook/Meta crawler (403) on custom domain
michaelkleinhenz
HOBBYOP

14 days ago

Our application uses the Instagram Content Publishing API, which requires Meta's servers to fetch images from our domain via image_url. Since recently, Meta's crawlers receive a 403 Forbidden when fetching any URL on our custom domain (www.socialpod.de), while the same URLs return 200 for regular clients.

Reproduction:

- Paste any public URL from our service (e.g. https://www.socialpod.de/api/uploads/1775420844419058563.png) into Meta's Sharing Debugger: https://developers.facebook.com/tools/debug/

- Result: Response Code: 403.

- The same URL returns 200 via curl and in browsers.

NOTE: the domain www.socialpod.de is a DNS CNAME referring directly to Railway infrastructure. There is no proxy involved.

Impact: All Instagram publishing via the Graph API is broken (error code 9004, subcode 2207052: "Failed to retrieve the media file at URI").

Ask: Could you allowlist Meta/Facebook crawler IPs or user-agents (facebookexternalhit) on the Fastly edge for our project?

Project: bab5e17c-24e2-4298-b641-721d0fc3768e

Service: d41e0c4a-b7dd-4f6c-9c51-7ba67eb481e4

Solved

2 Replies

Status changed to Awaiting Railway Response Railway 14 days ago


michaelkleinhenz
HOBBYOP

11 days ago

For whatever reason, it is working again since a few hours. I would really like to see consistency and transparency here. Someone is clearly patching around on the fastly settings. Railway, what is happening here? This is for many applications a mission critical issue.


We've traced this to Meta/Facebook's side. On our end, the request comes through and we respond with HTTP 200 — but Facebook's debugger reports a 403, which points to something happening after the response leaves us. Since the failure is on their end, it's not something we can fix from here. You'll want to raise this with Meta directly.


Status changed to Awaiting User Response Railway 9 days ago


Status changed to Solved ray-chen 9 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...