Draftcamp

AI crawler checker

Find out which of GPTBot, ClaudeBot, PerplexityBot and the rest your robots.txt actually lets in

Enter a URL to read its robots.txt

Twenty AI crawlers checked against your rules, grouped by what blocking each one actually costs.

Three decisions, not one switch

Most robots.txt advice about AI treats every bot as the same thing. It is not, and the difference decides whether you appear in AI answers at all.

Retrieval - blocking costs you citations

OAI-SearchBot, Claude-SearchBot, PerplexityBot fetch pages to answer a live question, and produce the citation link under the answer. Blocking these takes you out of AI search. For almost every business it is an error, and it is usually made by accident while trying to do something else.

User-triggered - a narrow cost

ChatGPT-User and Claude-User fetch your page when a person pastes the link into a chat. Blocking breaks that and nothing else.

Training - a real business decision

GPTBot, ClaudeBot, CCBot, Google-Extended collect corpora for future models. Blocking them has no effect on citation today. Many publishers block these deliberately. That is a choice, not a defect, and we report it as neither good nor bad.

The one that is not a crawler

Google-Extended never fetches anything. It governs whether content Googlebot already has can train Gemini, and blocking it does not touch Google Search. Teams block it fearing a ranking hit that cannot happen, or avoid it fearing a training opt-out they actually wanted.

To see whether the page is worth quoting once crawlers get in, run the AI visibility checker. To write the rules in the first place, use the robots.txt generator instead.

Common questions

Which AI crawlers matter most?
The retrieval ones: OAI-SearchBot, Claude-SearchBot and PerplexityBot. These fetch pages to answer a question someone is asking right now, and they are what produce the citation link under an AI answer. If those are blocked, you are not in AI results regardless of anything else on the page.
Is blocking GPTBot the same as blocking ChatGPT?
No, and this is the most consequential misunderstanding in the area. GPTBot collects training data for future models. OAI-SearchBot fetches pages to answer questions now. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
What is Google-Extended?
Not a crawler at all. It is an opt-out token that controls whether content Googlebot already fetched may be used to train Gemini. Blocking it has no effect whatsoever on Google Search indexing or ranking. Applebot-Extended works the same way for Apple.
I have no robots.txt. Is that bad?
No. A missing robots.txt means nothing is disallowed, which is the correct default for most sites. It is only worth adding one when you have something specific to exclude, or a sitemap to declare.
How current is the crawler list?
It comes from the same registry that powers the DraftCamp page-audit extension, and it is updated when vendors publish or rename tokens. Each entry records what blocking it actually costs, rather than treating every AI bot identically.