Every few weeks a new AI crawler shows up in the server logs, and the same question lands on the marketing team's desk: block it or allow it? The answer touches two separate things, whether your content trains someone else's model, and whether your business appears in the AI answers your customers now read first. Get it wrong and you can quietly delete yourself from ChatGPT, Perplexity, and Google's AI results while believing you protected your content. OpenAI is blunt about this in its official crawler documentation: sites that opt out of its search bot "will not be shown in ChatGPT search answers."
The decision is rules-based, not a guessing game. The crawlers that matter fall into two buckets, search and citation bots that put you in front of users, and training bots that feed model development. Once you can tell them apart, robots.txt gives you precise, per-bot control. This guide gives each crawler's exact user-agent token and a clear allow or block call.
Key Takeaways
- Search and citation bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot) decide whether you appear in AI answers. Blocking them silently removes you, so allow them.
- Training bots (GPTBot, Applebot-Extended, CCBot) control whether your content trains a model, and blocking them does not affect AI search visibility. Google-Extended is the exception: Google's documentation says it also covers grounding in the Gemini app, so blocking it can keep your pages out of Gemini app answers.
- Google-Extended and Googlebot are separate controls. AI Overviews use Google's normal index, so blocking Google-Extended does not hurt Search visibility, though it does affect grounding in the Gemini app.
- robots.txt is the control panel, and the user-agent token must be exact, or the rule does nothing.
- The real choice is a tradeoff: near-term visibility in AI answers versus long-term leverage over training and licensing.
The Two Kinds of AI Crawlers, and Why It Matters
Treating every bot as one group is the most expensive mistake we see. A search or citation bot fetches your page so an assistant can summarize and cite it in an answer. A training bot fetches it to help build a future model. They run under different user-agent names, so you can allow one and block the other. Block a training bot and you still appear when someone queries the assistant. Block a search bot and you vanish from that assistant even though you rank fine in classic results. Nothing breaks and no error appears, you simply stop being an option the AI can recommend. For any business investing in getting cited by ChatGPT and Perplexity, blocking the wrong bot undoes the entire effort.
OpenAI's Crawlers: Search, User Actions, Training, and Ads
OpenAI runs four bots that map onto the framework. Per its crawler docs, OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features," so allow it to be cited. ChatGPT-User fires when a person asks ChatGPT to visit a specific page, a user action rather than bulk crawling, so it belongs in the allow column, though as a user-initiated agent it may not fully follow robots.txt. GPTBot is the training crawler; OpenAI says disallowing it "indicates a site's content should not be used in training," and blocking it has no effect on ChatGPT search, which makes it the optional, strategic one. OAI-AdsBot only validates the safety of landing pages submitted as ads, not training, so allow it if you run ads inside ChatGPT.
Perplexity, Google, Apple, and Common Crawl
Perplexity mirrors the split. Per Perplexity's bot documentation, PerplexityBot surfaces and links websites in search results and is not used for training, while Perplexity-User is a visit triggered by a user's question. Perplexity recommends allowing PerplexityBot, so both are a clear allow. Googlebot is the only crawler behind Search, and Google confirms there is no separate crawler for AI features and no extra requirements to appear in AI Overviews or AI Mode, so never block it. Google-Extended is not an HTTP crawler at all, it is a robots.txt token that governs whether your content trains Gemini. Google's Google-Extended documentation is unambiguous: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." You can block Gemini training and keep full AI Overviews visibility.
Apple follows the same pattern. Applebot powers Spotlight, Siri, and Safari search, so allow it, while Applebot-Extended opts you out of training Apple's foundation models, and Apple confirms pages that disallow it can still appear in search results. CCBot (user-agent CCBot/2.0) belongs to Common Crawl, the nonprofit whose open corpus is a common training input for many AI labs. Blocking CCBot is a broad, one-line way to stay out of a widely reused dataset, with no effect on any live search product.
A Clear Allow or Block Recommendation
Here is our default for a business that wants maximum AI visibility while keeping leverage over training. Adjust it to your content strategy.
- Allow, always: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Googlebot, and Applebot. These are how you get seen, cited, and clicked. Blocking any of them removes you from that assistant.
- Allow if relevant: OAI-AdsBot, only if you run or plan ChatGPT ads and need landing pages validated.
- Optional block, mainly training: GPTBot, Applebot-Extended, and CCBot. Blocking these keeps your content out of training with no impact on AI answers or classic Search. Google-Extended also sits here with one caveat: it covers grounding in the Gemini app too, so blocking it can remove your pages from Gemini app answers.
In short, allow everything that cites you and decide deliberately about everything that only trains a model.
How to Configure robots.txt Correctly
robots.txt lives at the root of your domain and controls bots by user-agent token, one group per bot, and the token must match exactly, because a misspelled name matches nothing. To allow the search bots and block the training bots, give GPTBot, Google-Extended, Applebot-Extended, and CCBot their own groups with Disallow: /, and leave OAI-SearchBot, PerplexityBot, Googlebot, and Applebot untouched, since anything not disallowed is allowed. Two cautions save pain. A blanket User-agent: * Disallow: / blocks every well-behaved bot including the search crawlers and the classic index, so never use it here. And robots.txt is voluntary, respected by reputable crawlers but ignored by bad actors, so add server-level controls when you need hard enforcement. To set this once and verify it against every current bot, our answer engine optimization and SEO services configure and monitor crawler access.
The Real Tradeoff: Visibility Versus Training and Licensing
The training decision is strategic, not technical. Allowing training lets your expertise and data help shape models that may later answer questions without sending anyone to you. Blocking it preserves leverage, and some publishers block training deliberately to negotiate paid licensing rather than give content away. For a large content brand that leverage is worth protecting. For a local business or a store, the bigger risk is invisibility, so the priority is being allowed and readable. Either way, visibility depends on being easy for the search bots to parse, which is why structuring content for AI answer engines matters as much as permissions. Many teams also publish an llms.txt file to guide AI models toward their key pages, alongside a tuned robots.txt.
Blocking or allowing AI crawlers is not one decision, it is two. Allow every bot that cites you, because blocking a search crawler quietly removes your business from the answers customers now trust, then make a separate, deliberate call on the training bots. For an audit of which crawlers can reach your site today and a robots.txt tuned for AI visibility, talk to our team and we will map it for you.
Read More
- How to Get Your Brand Cited by ChatGPT and Perplexity
- Structuring Content for AI Answer Engines
- How to Build an llms.txt File: A Practical Guide
- Google AI Mode Is Now the Default: What It Means for Your Website
Frequently Asked Questions
Will blocking AI crawlers remove my site from ChatGPT and Perplexity?
It depends which crawler you block. Blocking the search and citation bots, OAI-SearchBot for ChatGPT or PerplexityBot for Perplexity, removes you from those assistants' answers. Blocking the training bots like GPTBot does not, because live answers are assembled by the search bots, not the training crawler.
Does blocking Google-Extended hurt my Google rankings or AI Overviews?
No. Google's documentation states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode use the normal index crawled by Googlebot, so your Search visibility stays intact. Blocking it does opt you out of Gemini training and of grounding in the Gemini app, so Gemini app answers may stop drawing on your pages.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's models, while OAI-SearchBot surfaces websites in ChatGPT's search features so they can be cited. Blocking GPTBot keeps your content out of training with no effect on visibility. Blocking OAI-SearchBot removes you from ChatGPT search answers.
Should a small business block any AI crawlers?
Usually not the search bots. For most local businesses and stores, the biggest risk is being invisible in AI answers, so allowing OAI-SearchBot, PerplexityBot, Googlebot, and Applebot is the priority. Blocking the training bots is optional, and only worth it if you have a specific reason to withhold content from training.
How do I control AI crawlers in robots.txt?
Add a separate group per bot using its exact user-agent token, and set Disallow only for the crawlers you want to block. Match tokens precisely, since a typo does nothing, and avoid a blanket wildcard disallow, which would block the search crawlers and your classic index too.
Want this handled for you? Our AI SEO services can help you rank on Google and get cited by ChatGPT, Gemini, Claude and Perplexity.
Aashirvad Kumar
Writes about SEO, digital marketing and growth at Optimize For SEO.