Should you block AI crawlers? Training, search, and agent bots, separated
GPTBot and ClaudeBot train models; OAI-SearchBot surfaces you in AI search; ChatGPT-User fetches for a live human. The robots.txt policy that says what you mean.
Don’t decide about “AI crawlers”; decide about three different kinds of visitor. Training crawlers (GPTBot, ClaudeBot) read your site to teach future models. Search crawlers (OAI-SearchBot, Claude-SearchBot) index it so AI search can send you visitors. User agents (ChatGPT-User, Claude-User) fetch pages because a human asked, right now. Blocking the first is a values call; blocking the second and third mostly means opting out of how a growing share of people find and read the web.
The three kinds of AI visitor, concretely
Both OpenAI and Anthropic now split their crawling by purpose and honor robots.txt per bot, which is what makes a per-purpose policy possible:
- Training:
GPTBot(OpenAI: used, in their words, to make their foundation models “more useful and safe”) andClaudeBot(Anthropic). Content fetched today shapes models shipped later. Blocking these costs you no traffic, and no presence inside future models either. - Search:
OAI-SearchBotsurfaces sites in ChatGPT’s search;Claude-SearchBotdoes the equivalent for Claude. These are the new referral pipes. Blocking them is the AI-era version of blocking Googlebot. - User-initiated:
ChatGPT-UserandClaude-Userfire when a person asks their assistant to read a page. OpenAI is explicit that because a human initiated the fetch, robots.txt rules may not apply to ChatGPT-User; Anthropic says Claude-User can be restricted. Either way, blocking this one tells a present, interested human’s assistant “no.”
A robots.txt that says what you actually mean
The common deliberate policy (no training, yes visibility) looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# search + user-initiated agents: allowed by default (no rules needed)Three mechanics worth knowing: rules apply per subdomain, OpenAI notes changes take ~24 hours to propagate, and robots.txt is a published request, not an enforcement mechanism. Reputable crawlers obey it, and the disreputable ones were never going to. If you want enforcement and receipts, Cloudflare’s AI Crawl Control (available on all plans) shows which AI crawlers visit, lets you allow/block them individually, and audits who actually respects your directives. It’s a different tool than Bot Fight Mode, which can’t make these distinctions at all.
Which policy fits your site
For a site that wants readers, allowing all three kinds is the stronger trade: as more of them arrive through AI answers or send an agent of their own to read for them, being the source an assistant can actually read and cite is worth more than whatever training-exclusion would protect. That math holds for a company blog with nothing paywalled; it inverts for businesses whose content is the product (publishers, courseware, paid research), where blocking training while keeping search visibility is entirely coherent. The mistake isn’t either policy; it’s the accidental one: a blanket block copied from a template in 2023, quietly deciding your AI-era distribution ever since.