Robots.txt in the age of AI crawlers: a field guide
Which user agents belong to which AI companies, what robots.txt can and cannot do, and example rules for allowing AI search while refusing AI training.
Robots.txt is one of the oldest conventions on the web and, until 2022, was never a formal standard. It is suddenly interesting again, because it is the main lever site owners have over AI crawlers. Used well, it lets you decide who may read what. Used carelessly, it can remove your site from Google.
What robots.txt is, and is not
The Robots Exclusion Protocol was finally published as an internet standard, RFC 9309, in September 2022. A crawler that respects it reads /robots.txt before crawling and follows the rules for its user agent.
Three things it is not:
- Not access control. It is a request. Well-behaved crawlers follow it; scrapers do not. Anything truly private needs a login.
- Not a way to remove pages from search. Google's documentation says robots.txt is mainly for managing crawler traffic, not for keeping a page out of Google. A blocked page can still appear in results if others link to it. To keep a page out, allow crawling and use a
noindextag. - Not retroactive. Blocking a crawler today does not remove what it collected before.
The user agents that matter now
AI companies generally separate crawlers that train models from those that fetch pages to answer a question. The distinction matters, because many sites want to be cited without being used for training.
- OpenAI:
GPTBot(training),OAI-SearchBot(ChatGPT search results),ChatGPT-User(fetches a page when a user asks). - Anthropic:
ClaudeBot(training), plus separate agents for search and user-requested fetches. - Google:
Google-Extendedis not a crawler but a robots.txt token that controls whether content crawled by Googlebot may be used for Gemini and Vertex AI models. It does not affect Google Search. BlockingGooglebotitself removes you from Search. - Perplexity:
PerplexityBotfor its index. - Common Crawl:
CCBot, whose public dataset is widely used for training. - Apple:
Applebotfor Siri and Spotlight, withApplebot-Extendedcontrolling use for Apple's AI training.
Each company documents its own agents; check their pages, because names change.
Example: allow AI search, refuse AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
A crawler follows the group that most specifically matches its name, so the specific groups override the general one.
Mistakes we see
- A leftover
Disallow: /from staging, copied to the live site. It happens more often than anyone admits. - Blocking CSS and JavaScript folders, which stops Google rendering the page properly.
- Blocking
/wp-admin/without allowingadmin-ajax.php, which some themes call from the front end. WordPress's default robots.txt allows it for that reason.
In Ironhelm
Crawlers and AI lets you allow or refuse each AI crawler by name, separates training crawlers from the ones that answer questions with a link back, and writes the matching rules into robots.txt with your sitemap. The staging guard keeps staging copies out of search engines on its own, so a blanket block never needs to exist on a site that will go live. AI visibility then shows which crawlers actually read your pages.
Sources
- IETF, RFC 9309: Robots Exclusion Protocol (September 2022)
- Google Search Central, Introduction to robots.txt
- Google, Google's common crawlers, including Google-Extended
- OpenAI, Overview of OpenAI crawlers
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Apple, About Applebot
Ironhelm Suite is one WordPress plugin for security, speed, SEO and analytics. See plans or tour the features.