What is Applebot-Extended?
Applebot-Extended is the crawler signal that controls whether Apple can use your pages to train its own AI models and Apple Intelligence features, separate from the Applebot crawler that indexes pages for Siri and Spotlight search results.
Apple runs two different bots under one shared crawl program. Applebot fetches pages so they can appear in Siri Suggestions, Spotlight search, and Safari's search results, the same job Googlebot does for Google Search. Applebot-Extended is a separate signal, checked only for a second purpose: whether Apple's foundation models, the ones behind Apple Intelligence, are allowed to train on that page's content. Blocking Applebot-Extended in robots.txt does not touch the first job.
The confusion runs the other direction from most AI crawler questions. Site owners who disallow Applebot-Extended sometimes expect their pages to also disappear from Siri or Spotlight results, and they don't, because Applebot keeps crawling under its own separate line. If the goal is to stop training use while staying visible in Apple's search surfaces, disallowing Applebot-Extended alone does exactly that, and adding a blanket Applebot block would be the mistake, not the fix.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether your content trains Gemini and grounds its answers. It is not a crawler and blocking it does not affect Google Search ranking.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- User agentA user agent is the name a client sends to identify itself when requesting a page. robots.txt rules are written against these names, which is why blocking AI crawlers means naming each one.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.