AI Crawlers vs AI Referrals: GPTBot, ClaudeBot and What to Actually Block

The Named Crawlers and What Each One Does

Group them by purpose rather than by vendor, because purpose is what determines the cost of blocking.

Training Crawlers

• GPTBot (OpenAI). Collects content for training future models. Blocking it removes your content from the next generation’s parametric knowledge. It has no effect on ChatGPT’s ability to browse to your page today.
• ClaudeBot (Anthropic). Anthropic’s training crawl. Same tradeoff shape as GPTBot.
• CCBot (Common Crawl). A nonprofit archive whose corpus feeds training at a very large number of organisations, including ones you have never heard of. The broadest single block available, and it also removes you from academic and research use.
• Bytespider (ByteDance). Widely reported as aggressive on crawl rate and inconsistent about honouring robots.txt. Teams that want to stop it do it at the WAF or CDN.

Retrieval and Search Crawlers

• OAI-SearchBot (OpenAI). Builds the index behind ChatGPT search results and the citations that appear beside answers. This is the one whose block directly costs you traffic. It does not contribute to training.
• PerplexityBot (Perplexity). Indexes pages so they can be surfaced and cited in Perplexity answers. Perplexity is the assistant that links most aggressively, so this crawler has an unusually high traffic-per-crawl value.

User-Initiated Fetchers

• ChatGPT-User (OpenAI). Fires when a person in ChatGPT asks something that requires visiting your page right now. Not a scheduled crawl. There is a human waiting on the response.
• Claude-User (Anthropic). The equivalent for Claude, fetching on behalf of a specific user in a specific conversation.

These two are the highest-intent requests on the list. Someone asked a question about your product and the assistant went to check. Block them and the model answers from stale memory, or says it cannot read the page, in front of a person who was actively evaluating you.

Permission Tokens, Not Crawlers

• Google-Extended. This fetches nothing. It is a robots.txt token that controls whether content Googlebot already crawled may be used for Gemini training and grounding. Disallowing it has no effect on your Google Search rankings or your inclusion in the Search index, which is the single most common misconception in this area.

Why Crawlers Never Show Up in Your Analytics

Client-side analytics runs a JavaScript snippet in the browser that sends a beacon to a collection endpoint. Most of the crawlers above request the raw HTML and parse it. They never build a DOM, never run the script, never fire the beacon. Your GA4 property and your Kissmetrics property therefore exclude them automatically, without any filter.

Worth stating plainly, because plenty of teams assume their traffic numbers are being inflated by AI bots. In a purely client-side setup, they are not.

Where They Do Appear

• Server access logs. Nginx, Apache, or your application framework. Complete and authoritative.
• CDN and edge logs. Cloudflare, Fastly, Vercel. Usually the best single source, because they see requests that never reach origin.
• WAF and bot-management dashboards. Where you find out what you are already blocking, often without having decided to.
• Server-side tracking, if you have not filtered. The actual failure mode. Move collection server-side for accuracy or privacy and forget user-agent filtering, and every crawler hit becomes a pageview in your product analytics.

The Rendering Exception

Some crawlers do render JavaScript, and more will as sites become more JS-dependent. A rendering crawler that executes your analytics snippet becomes a fake visitor with a datacenter IP, a 100% bounce rate and zero engagement. Same class of problem as classic referral spam, same detection approach. Our guide to GA4 bot and spam traffic covers the filtering side.

Telling a Crawler Hit From a Human Referral

The two produce almost no overlapping signals once you know what to look for.

Verify, Do Not Trust

A user agent string is a free-text field the client controls. Anyone can send GPTBot or claim to be Chrome. What actually verifies:

• Published IP ranges. OpenAI, Anthropic and Perplexity publish the ranges their crawlers operate from. A request claiming to be GPTBot from outside OpenAI’s ranges is something else pretending.
• Reverse DNS with forward confirmation. Reverse-resolve the IP, forward-resolve the hostname, check it matches. Same technique that verifies Googlebot, still the most robust.
• Asset ratio. Subresource requests per HTML document from the same client. A real browser pulls dozens. A ratio near one is machine traffic whatever the user agent says.
• Beacon correlation. An HTML response with no analytics event within a few seconds means the page was fetched but never rendered.

On the referral side, detection runs on the referrer hostname plus a utm_source fallback, since assistants strip referrers on native app surfaces. That mechanism is covered in full in tracking AI assistant traffic.

The Third Category

ChatGPT-User and Claude-User sit between the two columns above: crawlers technically, humans commercially, because a specific person is waiting on the answer. Do not file them with GPTBot. They are agents acting for a person and they deserve their own bucket and their own metrics, covered in analytics for AI agents.

The robots.txt Decision and Its Real Tradeoff

Split the question in two before answering either half.

Question One: Do You Want to Be in the Training Corpus?

What blocking GPTBot, ClaudeBot and CCBot costs is presence in the next model generation’s baseline knowledge. That baseline produces the unlinked mention: someone asks “what tools should I look at for X” and the model names you from memory without searching. Those mentions never appear in your referral data, and they are a meaningful share of how software gets shortlisted now.

The case for blocking is real for a narrow set of sites: original research you sell, proprietary datasets, licensed editorial, anything where the content is the product. The case against is strong for anyone whose content exists to be found, which is every marketing site. Blocking training crawlers on a marketing blog is asking not to be recommended.

Question Two: Do You Want to Be Citable Right Now?

Blocking OAI-SearchBot and PerplexityBot removes you from live answers with citations. Citations are what produce clicks, and clicks are the only part of this whole area you can measure in your own data. The cost is immediate and there is almost no scenario where a site that wants traffic should accept it.

Blocking ChatGPT-User and Claude-User is worse still. That is a person in the middle of an evaluation asking about you and getting told the page could not be read.

The Tradeoff Nobody Prices Correctly

The two questions are not fully independent. A model that has never seen your content in training is less likely to name you as a candidate at all, so the retrieval crawler is never sent to check. Blocking training crawlers reduces the number of opportunities for a citation without blocking citations directly. The effect is slow, indirect and impossible to measure cleanly, which is exactly why it gets ignored.

robots.txt Is Voluntary

It is a request, not a control. Well-behaved crawlers honour it. Others do not, and some scrape through third-party proxies carrying no identifying user agent at all. To prevent access rather than express a preference you need enforcement at the edge: WAF rules matched to verified IP ranges, rate limits, or authentication. A Disallow with nothing behind it stops the crawlers you were least worried about.

Rate Is a Separate Problem From Permission

A crawler costing you real money in bandwidth or origin CPU is a rate problem, and the answer is an edge rate limit rather than a block. Crawl-delay is inconsistently supported and should not be relied on. Rate-limiting keeps you in the index while capping the cost.

Three Practical Postures

Open

Allow everything. Correct for essentially every marketing site, documentation site, and content business whose growth depends on being found. If you are unsure, this is the default and the burden of proof is on blocking.

Selective

Allow retrieval and user-initiated agents, block training crawlers. This is the posture for publishers and research businesses that want the traffic without contributing the corpus. It looks like this:

# Training crawlers: no
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Retrieval and citation: yes
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-initiated fetches: yes, a real person is waiting
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Gemini training and grounding, does not affect Google Search
User-agent: Google-Extended
Disallow: /

This posture is a bet: reduced presence in future model knowledge in exchange for keeping today’s citations. Revisit it, and enforce it at the edge if it matters commercially, because the training crawlers you block here are the ones most likely to route around you.

Closed

Block everything. Only defensible for paywalled content, licensed material, or a business whose entire value is exclusivity of information. Enforce it at the edge, because robots.txt alone will not deliver it.

Key Takeaways