What is Claude-SearchBot, and how is it different from ClaudeBot?

· Edited by Chris Dolan

Claude-SearchBot is the crawler Anthropic uses to improve the search results Claude gives its users. ClaudeBot is a different crawler, which collects content that may be used to train Anthropic's models. A site can allow one and refuse the other, because Anthropic lists them as separate robots. Every statement below was checked against Anthropic's own pages on 9 October 2026.

The practical point is this: blocking ClaudeBot asks Anthropic to leave your future pages out of training, and blocking Claude-SearchBot may reduce how visible and how accurate your site is in Claude's search results. The two choices are separate, and a robots.txt file that refuses both makes both.

What Claude-SearchBot is

Anthropic's help article on how it crawls the web, dated 7 April 2026, lists three robots: ClaudeBot, Claude-User and Claude-SearchBot. It describes the third in one line: “Claude-SearchBot navigates the web to improve search result quality for users.”

Web search in Claude is what makes this matter. Anthropic introduced it as a preview on 20 March 2025, and an update to the announcement dated 27 May 2025 says: “Web search is now available globally on all Claude plans.” On the pages read for this article Anthropic does not say which search index stands behind Claude's answers, so this article does not either. It repeats only what Anthropic says about its own crawler.

ClaudeBot, Claude-SearchBot and Claude-User: three robots, three purposes

The help article gives each robot a purpose and says what happens when a site turns it away.

  • ClaudeBot collects web content that could contribute to training Anthropic's models. Restricting it signals that the site's future material should be left out of training.
  • Claude-SearchBot analyzes content to make search responses more relevant and accurate. Anthropic says disabling it prevents its systems from indexing the site for search, which may reduce the site's visibility and accuracy in search results.
  • Claude-User fetches a page when a person using Claude asks a question that needs it.

Anthropic gives a reason for the split: it uses different robots so that site owners can see what each is for and choose between them.

Where Claude-User fits

Claude-User is not a crawler in the usual sense. Anthropic's help article on using web search describes the kind of request it serves: “Claude can also retrieve content directly from web pages when provided with specific URLs”. Disabling Claude-User stops that, which Anthropic says may reduce a site's visibility for user-directed web search.

On robots.txt, Anthropic's position is that its robots follow it. Search Engine Journal summarized it this way: “Anthropic says all three of its bots honor robots.txt, including Claude-User.” Anthropic's developer documentation for its web fetch tool points the same way: it lists “Anthropic-side restrictions, such as private addresses, robots.txt” among the reasons a fetch can be refused. Our guide to what AI search shows about your business explains why visits made for a user are counted separately from crawling.

Blocking or slowing a robot in robots.txt

To keep a robot off the whole site, Anthropic's article gives this rule, using ClaudeBot as its example:

User-agent: ClaudeBot
Disallow: /

Three things are worth knowing before you rely on it.

  • It applies per subdomain. Anthropic asks for the rule in the robots.txt file of every subdomain you want to opt out.
  • You can slow a robot instead of blocking it. Anthropic supports the Crawl-delay line, which is not part of the standard, and shows it with a value of 1.
  • The name is not case-sensitive. The standard that defines robots.txt, RFC 9309, says: “Crawlers MUST use case-insensitive matching” for the name on the User-agent line, so ClaudeBot and claudebot are one rule.

Anthropic's examples all use ClaudeBot. To limit Claude-SearchBot or Claude-User, write the same lines under that robot's name. A robots.txt block is also the first thing to rule out when a site is missing from AI answers, and the technical accessibility section of our audit checklist starts there.

How to recognize Anthropic's traffic

Any program can call itself ClaudeBot, so the name in a log line settles nothing. Anthropic publishes the addresses its robots use in one file, bots.json, and says of an address on that list: “it indicates that the crawler is coming from Anthropic”. Unlike some other operators, it publishes a single list for all its robots, not one per robot.

Anthropic warns against using the list the other way round. Blocking its addresses, it says, may not work correctly or reliably guarantee an opt-out, because it stops the robots reading your robots.txt file. Use the list to recognize traffic and robots.txt to refuse it.

What changed in 2025 and 2026

Anthropic's article shows a single date and no change history, so the first date below comes from trade press that noticed the edit.

  • February 2026. Anthropic expanded the article from one named robot to three. Search Engine Roundtable reported on 25 February: “Anthropic has three primary bots and they include ClaudeBot, Claude-User and Claude-SearchBot.”
  • 7 April 2026. The date the article shows today, so it has been edited again since.

Older articles that describe a single Anthropic crawler were written before these edits, and anything they say about blocking it should be checked against the current article.

Sources

All eight were read on 9 October 2026.

  1. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center, 7 April 2026.
  2. Claude web search now available globally on all plans, Anthropic, 20 March 2025, with an update of 27 May 2025.
  3. Enable and use web search, Claude Help Center.
  4. Anthropic's Claude Bots Make Robots.txt Decisions More Granular, Search Engine Journal, 25 February 2026.
  5. Web fetch tool, Claude Platform documentation (no date shown).
  6. RFC 9309: Robots Exclusion Protocol, IETF, September 2022.
  7. bots.json, Anthropic's published addresses for its robots.
  8. Anthropic Updates Its Crawler Documentation: ClaudeBot, Claude-User & Claude-SearchBot, Search Engine Roundtable, 25 February 2026.