Should You Block AI Crawlers Like GPTBot and ClaudeBot on Your Website?

By Bill Stilwell · Last reviewed October 16, 2026

The short answer

Most local businesses should allow the search crawlers that put them in artificial intelligence (AI) answers, then decide separately whether to allow crawlers that collect content for AI training. Each company uses different crawler names for search and for training, and you control them in your robots.txt file. Blocking a training crawler like GPTBot or Google-Extended does not, by itself, remove you from ChatGPT search or Google Search.

What this means

A crawler is a program that visits web pages and reads them. Your robots.txt file is a short text file at the root of your site that tells crawlers what they can visit. Each crawler has a name, called a user agent.

Here’s what the major companies say their crawlers do:

Company	Crawler name	What it’s for
Google	Googlebot	Google Search, including AI Overviews and AI Mode
Google	Google-Extended	Whether content trains Gemini and grounds answers in Gemini Apps and Vertex AI. Google says it doesn’t affect Search inclusion or ranking
OpenAI	OAI-SearchBot	Showing sites in ChatGPT search
OpenAI	GPTBot	Collecting content for AI training
OpenAI	ChatGPT-User	Visits a person asks ChatGPT to make. OpenAI says robots.txt rules may not apply
Anthropic	Claude-SearchBot	Indexing pages to improve search results
Anthropic	ClaudeBot	Collecting content for AI training
Anthropic	Claude-User	Fetching pages when a person asks Claude a question
Perplexity	PerplexityBot	Perplexity search results. Perplexity says it’s not used to train AI models
Perplexity	Perplexity-User	Visits a person asks for. Perplexity says it generally ignores robots.txt
Apple	Applebot	Siri, Spotlight, and Safari search
Apple	Applebot-Extended	Whether Applebot’s data trains Apple’s AI models. It doesn’t crawl on its own

Microsoft Bing works differently. Instead of a separate AI crawler, Bing announced in 2023 that two meta tags control use in its chat answers: NOCACHE limits answers to your link, title, and snippet, and NOARCHIVE keeps the page out of chat answers and training. Pages with either tag still show in regular Bing results.

Say a landscaper wants to be found in ChatGPT and Perplexity but would rather not have their articles used to train AI models. They allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and block GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended.

Why it matters

If you block the wrong crawler, you can disappear from places customers look. A blanket “block all AI bots” rule from a security plugin can quietly remove you from ChatGPT search or Perplexity.

On the other hand, some owners don’t want their writing used to train AI models. The training and search crawlers are separate for most companies, so you can make that choice without giving up search visibility.

Who this applies to

Anyone with a website. If you use a website builder or a host like a content delivery network (CDN), check its bot settings too. Some block AI crawlers by default.

How to do it

Open your robots.txt. Visit yoursite.com/robots.txt. If there isn’t one, your platform may let you create it (see 19.5.1 Crawl Access and robots.txt).
Make your choice in writing. Decide which search crawlers to allow and whether to allow training crawlers. Write it in your website records.
Add rules by crawler name. For example, to block training but allow search for OpenAI, add a group for GPTBot with “Disallow: /” and leave OAI-SearchBot allowed.
Don’t block Googlebot or Bingbot unless you want out of search completely.
To stay out of Google AI Overviews, don’t use Google-Extended. Google says Googlebot rules control Search. Google’s tools for limiting AI Overviews are snippet controls like nosnippet and, since 2026, a “Search generative AI control” setting in Search Console. Opting out costs you the traffic from those features.
Check your host and CDN settings. Look for “bot protection” or “AI bot blocking” switches that override robots.txt.
Wait and check. OpenAI and Perplexity say changes take about 24 hours to take effect.
Review twice a year. New crawlers appear, and companies change what each one does.

What proof to keep

A dated copy of your robots.txt file
A short note of your decision: which crawlers you allow and why
Screenshots of host or CDN bot settings

Store them in your website records folder with your access inventory.

Common mistakes

Blocking “all bots.” That includes Googlebot and Bingbot, which removes you from search.
Thinking Google-Extended controls AI Overviews. Google says it doesn’t affect Search.
Thinking robots.txt is a lock. Well-behaved crawlers follow it. Some user-triggered fetchers say they may not. Private content belongs behind a login.
Using noindex with a blocked crawler. OpenAI notes its crawler has to be allowed in to see a noindex tag.
Forgetting the firewall. Your CDN may be blocking crawlers that your robots.txt allows.

When to get help from a pro

A web developer or search engine optimization (SEO) specialist: if you’re not sure what your host is blocking, or your site suddenly stopped showing in AI answers. Questions to ask:

Which crawlers are blocked right now, in robots.txt and at the server or CDN?
Will this change affect our regular Google or Bing results?

Bill’s note

[Bill: add your opinion here. For example, what you recommend to clients about training crawlers and why, or a site you found that was blocking every AI crawler without the owner knowing.]

If you’d rather have this done for you, SiteSmarty sets up and manages websites for local businesses, including crawler settings.

How I did this for SiteSmarty

[Date I did this:]
[What I actually did, step by step: which crawlers I allowed and blocked on SiteSmarty and BillStilwell.com, and why]
[What it cost and how long it took:]
[What surprised me or what I’d do differently:]
[Screenshot or photo to add, with private info blurred: the robots.txt file and the host bot settings]
[Result so far:]

Questions people ask

What is Google-Extended? A robots.txt name that controls whether Google uses your content to train Gemini and ground some Gemini answers. Google says it doesn’t affect Google Search.

Does blocking GPTBot remove me from ChatGPT? Not from ChatGPT search. OpenAI says GPTBot is for training and OAI-SearchBot is for search, and each setting is separate.

Do AI crawlers follow robots.txt? The main search and training crawlers say they do. Some user-triggered fetchers, like ChatGPT-User and Perplexity-User, say they may not.

How do I keep my site out of Google AI Overviews? Use snippet controls or the Search Console generative AI setting. Both reduce the traffic you get from those features.

How fast do changes take effect? OpenAI and Perplexity say about 24 hours.

Audit check

Question: Does the robots.txt file and host setup match a written decision about which AI search and training crawlers are allowed?
Pass looks like: Googlebot and Bingbot allowed. AI search crawlers allowed if the business wants AI visibility. Training crawlers allowed or blocked on purpose. Host and CDN settings match.
Fail looks like: All bots blocked. A firewall blocking crawlers the robots.txt allows. Google-Extended used to try to block AI Overviews. No record of the decision.
Evidence to collect: Dated robots.txt copy, the written decision, and screenshots of host or CDN bot settings.

Where this fits

Do this first: 19.5.1 Crawl Access and Robots.txt, 19.13.1 How AI Tools Choose Sources

Do this next: 19.13.3 Bing, IndexNow, and AI Search

Sources

Google Search Central, "Overview of Google crawlers and fetchers (common crawlers)" (checked October 16, 2026)
Google Search Central, "AI features and your website" (checked October 16, 2026)
Search Console Help, "Search generative AI control" (checked October 16, 2026)
OpenAI, "Overview of OpenAI Crawlers" (checked October 16, 2026)
OpenAI Help Center, "Publishers and Developers FAQ" (checked October 16, 2026)
Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (checked October 16, 2026)
Perplexity, "Perplexity Crawlers" (checked October 16, 2026)
Apple Support, "About Applebot" (checked October 16, 2026)
Bing Webmaster Blog, "Announcing new options for webmasters to control usage of their content in Bing Chat" (September 22, 2023) (checked October 16, 2026)

This page explains general rules and Bill's own experience. It is not legal, tax, or insurance advice. Talk to a qualified professional about your situation.