Draftcamp

Robots.txt generator

Build a robots.txt that treats AI training, retrieval and user-fetch crawlers as the separate decisions they are

Runs entirely in your browser - your text is never uploaded

Search engines

One path per line

Ignored by Google

AI crawlers

Three separate decisions, not one. Blocking is off by default.

Retrieval (citation-eligible)

These produce citations. Blocking them removes you from AI answers entirely - almost always a mistake.

User-triggered fetch

These fetch your page when someone pastes the link into a chat. Blocking breaks that, and nothing else.

Model training

These collect training data for future models. Blocking has no effect on whether you get cited today. A legitimate business decision either way.

robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?s=

Upload to the root of your domain, at /robots.txt exactly. Each subdomain needs its own.

Blocking AI crawlers is three decisions, not one

Almost every robots.txt generator offers a single "block AI bots" checkbox. That checkbox conflates three completely different outcomes, and the most common result is a site that meant to opt out of model training and accidentally deleted itself from AI search.

Retrieval crawlers - blocking these costs you citations

OAI-SearchBot, Claude-SearchBot and PerplexityBot are the ones that matter here. These fetch pages to answer a question someone is asking right now, and they are what produce the citation link under an AI answer. Block them and you are not in the answer. For almost every business this is a straightforward mistake.

User-triggered fetchers - a narrow, specific cost

ChatGPT-User, Claude-User. These fetch your page when a person pastes your link into a chat and asks about it. Blocking them breaks that flow and nothing else. Reasonable to block if your pages are gated; otherwise pointless.

Training crawlers - a legitimate business decision

GPTBot, ClaudeBot, CCBot, Google-Extended. These collect corpora for training future models. Blocking them has no effect on whether you get cited today. Plenty of publishers block these deliberately, and that is a defensible choice - not a defect, which is why no tool should flag it as one.

The one that is not a crawler

Google-Extended never fetches anything. It is an opt-out token controlling whether content Googlebot already has can train Gemini. Blocking it does not affect Google Search indexing or ranking at all. Applebot-Extended is the same idea for Apple.

Once the file is live, confirm what it actually permits with the rest of the free tools as well.

Common questions

Should I block AI crawlers?
It depends entirely on which kind, and lumping them together is how people accidentally remove themselves from AI search. Blocking retrieval bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot means you cannot be cited in AI answers. Blocking training bots like GPTBot and CCBot has no effect on citation at all - it only declines future model training. Most people who say "block AI" mean the second and accidentally do the first.
What is the difference between Google-Extended and Googlebot?
Google-Extended is not a crawler. It is an opt-out token controlling whether content Googlebot already fetched can be used for Gemini training. Blocking it does not affect your Google Search indexing or ranking in any way. Applebot-Extended works the same way for Apple.
Does robots.txt stop a page being indexed?
No, and this is the most common robots.txt mistake. Disallow stops crawling, not indexing. A page blocked in robots.txt can still appear in results - usually with no description - if other pages link to it. To keep something out of the index you need a noindex meta tag, and that requires the page to be crawlable so the tag can be read.
Where does the file go?
At the root of the domain, exactly: https://example.com/robots.txt. It applies per protocol, host and port, so every subdomain needs its own. Crawlers do not look anywhere else and will not find it in a subdirectory.
Does Crawl-delay work?
Not for Google, which ignores it entirely - use the crawl rate setting in Search Console instead. Bing and Yandex do respect it. It is here for those, but on most sites it should be left empty.