What is meta-externalagent ?
meta-externalagent is a user-agent used by Meta Platforms, Inc. (formerly Facebook) to fetch web content. It has been observed accessing publicly available resources, particularly in contexts related to the development or refinement of AI models. Meta confirmed in August 2023 that this user-agent is part of their efforts to collect data to improve generative AI systems. Unlike traditional bots like Facebook Crawler (used for previews), meta-externalagent is tied to machine learning data intake.
Who is operating meta-externalagent ?
The agent is operated by Meta Platforms, Inc., the parent company of Facebook, Instagram, and WhatsApp. Meta disclosed that this user-agent is used to gather content to support their foundation model efforts. Official disclosure about its usage was made via updates to their crawler policy and AI documentation, although no detailed technical page has been published as of now.
Why you should be interested in meta-externalagent ?
This agent is relevant because it fetches data that may be reused in Meta’s generative AI pipelines. That includes language and vision models deployed across Meta products. If you are concerned with how your content is used, particularly in the context of AI training, this user-agent should be explicitly considered in your robots.txt or server-side blocking rules. According to multiple bot monitoring services, its adherence to robots.txt varies depending on configuration.
How to block meta-externalagent ?
1. robots.txt File:
Add the following rule to your robots.txt file
# block meta-externalagent User-agent: meta-externalagent Disallow: /
2. Server Filtering:
Use web server rules to block the user-agent string “meta-externalagent”.
3. Log Monitoring:
The agent can be identified in logs by its user-agent string and often operates under IP ranges associated with Meta’s infrastructure.
About the bot
Owner: Meta Platforms, Inc.
Owner URL: about.meta.com
Bot URL: transparency.fb.com
Bot User Agent: Mozilla/5.0 (compatible; meta-externalagent/1.0; +https://about.meta.com)
Respects robots.txt: Partially
