Guides

Is your website blocking ChatGPT? How to check

By Patrick Colton. Published October 17, 2026.

Type your web address into a browser with /robots.txt on the end and read what comes back. If you see Disallow: / under User-agent: OAI-SearchBot or under User-agent: * (which covers every bot the file does not name), your site is telling ChatGPT search to stay out. A clean file is only half the check. A firewall at your host or at Cloudflare can turn the same bots away before they read a word.

What is robots.txt?

It is a plain text file at the top level of your website. Google describes it as the file that tells crawlers which pages on your site they may visit. Crawlers are the programs that search engines and AI companies send out to read web pages.

The file is made of groups. Each group starts with a User-agent: line that names a crawler. Under it, Allow: and Disallow: lines say where that crawler may and may not go. Disallow: / means the whole site.

The file is only a request. The standard that defines it says its rules are not access control. Nothing in it stops a crawler that chooses to ignore it.

How do I look at mine?

Go to yourdomain.com/robots.txt in any browser, using your real address.
Use the version your site loads on, with or without the www. Google reads a separate file for each.
Read what comes back.

A "not found" page. You have no file. Under the standard, crawlers may then visit every page, which is fine if you want to be found.
A server error or a page that never loads. Under the standard, a crawler that gets a server error on this file must treat the whole site as off limits.
Plain text. Find each User-agent: line and read the Disallow: lines under it.
Cloudflare lines. If your site runs through Cloudflare, you may see lines between # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. Cloudflare adds those itself when some of its bot settings are on. Crawlers read them like the rest of the file.

Which crawlers matter and what does each one feed?

Most of the big AI companies run separate crawlers for separate jobs. One kind collects pages to train their models. Another reads pages so the product can find and link to them when someone asks a question. A third fetches a page at the moment a person asks about it. Blocking the first kind is a choice about training. Blocking the other two can keep you out of the answers.

OpenAI (ChatGPT)

OAI-SearchBot puts websites in ChatGPT's search results. OpenAI says a site that blocks it will not be shown in ChatGPT search answers, though it can still show up as a plain link. It is the crawler to check first.
GPTBot collects pages that may be used to train OpenAI's models. Blocking it tells OpenAI not to train on your content. OpenAI says the settings are independent, so you can allow OAI-SearchBot and block GPTBot.
ChatGPT-User visits a page when someone using ChatGPT asks something that needs it. OpenAI says robots.txt rules may not apply to it, because a person started the request.

Anthropic (Claude)

Claude-SearchBot reads and indexes pages for Claude's search results. Anthropic says blocking it may cut your visibility in those results.
Claude-User fetches a page when a Claude user's question needs it. Blocking it stops Claude from pulling up your pages for those questions.
ClaudeBot collects content that may be used to train Anthropic's models. Blocking it asks Anthropic to leave your future content out of training.
Anthropic says all three follow robots.txt.

Perplexity

PerplexityBot finds and links websites in Perplexity's search results. Perplexity says it is not used to collect content for training AI models.
Perplexity-User visits a page when a user's question needs it. Perplexity says it generally ignores robots.txt, because a person asked for the page.

Google

Googlebot crawls for Google Search and every Search feature. That includes AI Overviews and AI Mode. Google says AI is built into Search and that your Googlebot rules are how you control crawling for Search.
Google-Extended is not a crawler. It is a robots.txt name that says whether Google may use your pages to train Gemini models and to ground answers in the Gemini app and in Google Cloud's Vertex AI. Google says Google-Extended "does not impact a site's inclusion in Google Search" and is not a ranking signal. Blocking it does not take you out of AI Overviews or AI Mode.

Microsoft (Bing and Copilot)

Bingbot is Bing's standard crawler. Bing's guidelines cover both Bing search and Copilot. They say both rely on the same crawling, indexing and ranking as traditional search. They list blocking Bingbot in robots.txt as something to avoid.

Apple (Siri, Spotlight and Safari)

Applebot powers search in Spotlight, Siri and Safari. Apple says what it crawls may also be used to train Apple's AI models.
Applebot-Extended is not a crawler either. Block it to keep your pages out of Apple's model training. Apple says pages that block it can still be included in search results.
If your file has rules for Googlebot and none for Applebot, Apple says Applebot follows the Googlebot rules.

Which lines would block them?

Put Disallow: / under a User-agent: line naming one of the search crawlers above and you have asked it to stay off the whole site. Blocking Googlebot this way also blocks Applebot if the file has no Applebot group.

The line to look at twice is User-agent: * with Disallow: / under it. That blocks every crawler the file does not name in a group of its own. It is sometimes added while a site is being built. If nobody takes it out at launch, it keeps blocking.

GPTBot, ClaudeBot and Applebot-Extended only opt you out of training, by each company's account. Google-Extended also reaches the Gemini app's grounded answers, but not Google Search. Blocking the training crawlers is your call. A file for an owner who wants to be found but not trained on can be four short groups. Each group is two lines, with a blank line between groups:

User-agent: GPTBot then Disallow: /
User-agent: ClaudeBot then Disallow: /
User-agent: Applebot-Extended then Disallow: /
User-agent: * then Allow: /

We left Google-Extended out, because blocking it also keeps your pages out of the Gemini app's grounded answers.

Can a firewall block bots even when robots.txt allows them?

Yes. A firewall does not ask. It refuses the request before robots.txt comes into it. Hosts and CDNs (services like Cloudflare that sit in front of your site) can both do this. Google's own checklist for its AI features tells site owners to make sure crawling is allowed in robots.txt and by any CDN or hosting setup.

Cloudflare deserves a close look, because its AI bot settings changed this year.

Three groups of AI bots. Cloudflare sorts them into Search (crawlers that index content to answer questions later), Agent (bots acting for a person in real time, such as chat fetch bots) and Training. Each group can be set to Allow, to Block on all pages or to Block on pages with ads. The setting is in the Cloudflare dashboard under Security Settings, then Configure AI bot policies.
A new default. Cloudflare says that from September 15, 2026, new domains start with Training and Agent blocked on pages its systems detect as showing ads. Search stays allowed. That default blocks only on those pages. Look for any group set to Block on all pages.
Training blocks reach further. Cloudflare says blocking Training can also catch crawlers that do both search and training.
Crawler by crawler. Cloudflare's AI Crawl Control lets you allow or block each named AI crawler. A block there is enforced by a firewall rule, not by robots.txt. Its Crawlers tab shows each crawler's requests and how many failed.

Other hosts and firewalls can do the same thing. Perplexity's documentation says a site behind a firewall may need to allow its bots explicitly. OpenAI and Perplexity both publish the IP addresses their crawlers use, so a firewall can let the real ones in.

What should I ask my web person or host?

What does our live robots.txt block, line by line?
Is the site behind Cloudflare or another firewall? If it is Cloudflare, what are Search, Agent and Training set to? Is any crawler set to Block in AI Crawl Control?
Do the firewall or server logs show OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot or Applebot being refused?
If we allow bots by IP address, are we using the lists OpenAI and Perplexity publish?
In Google Search Console, under Settings, is Search generative AI set to include our site? Include is the default, so someone would have had to change it.

After a change, give it a day. OpenAI and Perplexity both say a robots.txt change can take up to about 24 hours to show up on their side.

What does unblocking do?

It makes your pages readable to the systems that write the answers. It does not make any AI name your business. Google says indexing and serving are not guaranteed, even for pages that meet all of its requirements. Whether an AI names you then depends on what it finds on those pages and in other sources.

We check the live file and the firewall on every Tune-Up and let in the crawlers that feed AI answers. Every job starts with the GEO Audit: $500, credited in full toward any package bought within 30 days.

Sources. OpenAI, "Overview of OpenAI Crawlers," developers.openai.com/api/docs/bots. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" support.claude.com/en/articles/8896518. Perplexity, "Perplexity Crawlers," docs.perplexity.ai/docs/resources/perplexity-crawlers. Google Crawling Infrastructure, "Google's common crawlers," developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers. Google Crawling Infrastructure, "How Google interprets the robots.txt specification," developers.google.com/crawling/docs/robots-txt/robots-txt-spec. Google Search Central, "Introduction to robots.txt," developers.google.com/search/docs/crawling-indexing/robots/intro. Google Search Central, "AI features and your website," developers.google.com/search/docs/appearance/ai-features. Google Search Central, "Optimizing your website for generative AI features on Google Search," developers.google.com/search/docs/fundamentals/ai-optimization-guide. Search Console Help, "Search generative AI control," support.google.com/webmasters/answer/16908024. Bing Webmaster Tools, "Which crawlers does Bing use?" bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0. Bing Webmaster Tools, "Webmaster Guidelines," bing.com/webmasters/help/webmaster-guidelines-30fba23a. Apple Support, "About Applebot," support.apple.com/en-us/119829. Cloudflare Docs, "Block AI Bots," developers.cloudflare.com/bots/additional-configurations/block-ai-bots. Cloudflare Docs, "Bots changelog," developers.cloudflare.com/bots/changelog. Cloudflare Docs, "robots.txt setting," developers.cloudflare.com/bots/additional-configurations/managed-robots-txt. Cloudflare Docs, "Manage AI crawlers," developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers. IETF, RFC 9309, "Robots Exclusion Protocol," rfc-editor.org/rfc/rfc9309.