Many sites block "AI bots" as one group. That often blocks the crawlers that let ChatGPT, Claude, Perplexity and Apple show and link your pages, not only the ones that collect training data. The major vendors now use separate robots.txt names for each job, so you can choose.
Checked against each vendor's own documentation on 13 October 2026. Vendors change these pages; follow the source links before you rely on a line.
| Name in robots.txt | Vendor | What the vendor says it does | Source |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. | OpenAI bots |
ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks a question. Not used for automatic crawling and not used to decide search inclusion. OpenAI says robots.txt rules may not apply because the visit is user-initiated. | OpenAI bots |
Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results. Blocking it prevents your content from being indexed for search. | Anthropic crawlers |
Claude-User | Anthropic | Fetches pages when a Claude user asks a question. | Anthropic crawlers |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. | Perplexity crawlers |
Perplexity-User | Perplexity | Visits a page when a Perplexity user asks a question. Perplexity says this fetcher generally ignores robots.txt because a user requested it. | Perplexity crawlers |
Applebot | Apple | Powers search in Spotlight, Siri and Safari. Apple says its data may also give context to AI-generated answers in Siri and Search. | About Applebot |
OpenAI also lists OAI-AdsBot, which only checks landing pages submitted as ads on ChatGPT. It matters only if you advertise there.
| Name in robots.txt | Vendor | What blocking it does, per the vendor | Source |
|---|---|---|---|
GPTBot | OpenAI | Signals that your content should not be used to train OpenAI's foundation models. OpenAI says each setting is independent, so you can block GPTBot and still allow OAI-SearchBot. | OpenAI bots |
ClaudeBot | Anthropic | Signals that your future content should be excluded from Anthropic's training data. | Anthropic crawlers |
Applebot-Extended | Apple | Opts your content out of training Apple's generative models. Applebot can still crawl, and your site stays discoverable in Spotlight, Siri and Safari. | About Applebot |
CCBot | Common Crawl | Stops Common Crawl, a non-profit that publishes an open archive of web pages anyone can download. Common Crawl warns that some crawlers falsely identify as CCBot. | CCBot |
Google-Extended | Not training-only. Google says it controls use for training Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect Google Search inclusion or ranking. Blocking it can keep your site out of Gemini app answers. | Google crawlers |
Adapt this to your own site. It only adds AI groups; keep your existing rules for other crawlers.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Applebot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
We left Google-Extended out on purpose: blocking it also affects Gemini app answers. Add User-agent: Google-Extended with Disallow: / only if keeping out of Gemini training matters more to you than appearing there.
Our free check reads the robots.txt and page source you give it, in your browser, and tells you whether any of the search crawlers above are blocked. Nothing is uploaded.