Is your website blocking AI without you knowing? robots.txt and the crawlers of ChatGPT, Google, Claude, Perplexity and Apple (2026)

Updated: 11 November 2026

An AI assistant can only cite your website if its crawler can read it. Many hotel websites block crawlers without the owner knowing: because of a template, a security plugin or a provider setting. This guide explains which crawler each company uses, what for, and how to check it in five minutes, based on official documentation (sources at the end).

The crawlers that make you appear (don't block them)

Each company separates its search crawler from its training crawler. To appear in answers, these must be allowed in:

ChatGPT: OAI-SearchBot. OpenAI says it is the one that surfaces websites in ChatGPT's search features.
Google (AI Overviews and AI Mode): Googlebot. Google says there are no additional requirements to appear in its AI features: the page just needs to be crawlable and indexable.
Claude: Claude-SearchBot, which according to Anthropic improves search result quality.
Perplexity: PerplexityBot, which according to Perplexity surfaces and links websites in its results and is not used to train models.
Siri, Spotlight and Safari: Applebot.
Bing (and whatever relies on Bing): Bingbot.

Training crawlers (your decision)

Blocking them does not remove you from results, according to the companies themselves:

GPTBot (OpenAI): blocking it signals that your content should not be used to train models.
Google-Extended: Google says it does not affect your inclusion in Google Search and is not a ranking signal.
New (30-10-2026): Google announced that in 2027, together with other AI providers, it will support a shared robots.txt signal to say "don't use my site for training" (based on the IETF AIPREF work). It does not work yet: for now, use each crawler's name.
ClaudeBot (Anthropic) and Applebot-Extended (Apple): Apple says pages that disallow it can still be included in search results.
There are also crawlers that only visit when a user asks (ChatGPT-User, Claude-User, Perplexity-User): if you block them, a traveller who asks the assistant to open your website won't be able to.

How to check it in five minutes

Open your-site.com/robots.txt in a browser. If you see “User-agent: *” followed by “Disallow: /”, you are blocking everyone.
Look for the names above. A “Disallow: /” line under OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot or Applebot removes you from that assistant.
If your website uses Cloudflare: in July 2025 Cloudflare announced it would ask every new domain whether to allow AI crawlers. Its current documentation says they are allowed by default on new domains, but if your site was set up earlier, check the AI crawler and agent access settings in the dashboard.
Ask whoever runs your website or booking engine whether a security plugin or bot blocker blocks on its own. robots.txt is useless if the crawler never reaches the page.

A simple example

If you want to be found and don't mind the rest, simply don't block anyone. If you block training, explicitly keep the search crawlers open: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot.

How iQuota helps

The official page iQuota publishes for each property lets AI search crawlers in, is plain text and carries structured data. iQuota's connector also gives compatible assistants real availability and prices with direct booking, commission-free and with payment at the property. It does not guarantee that an AI will recommend you.

Want a booking page AI can read and a connector with your real prices, commission-free? The first 30 hotels join free until 30 June 2027.

See the Founding Hotels programme

Frequently asked questions

If I block GPTBot, do I disappear from ChatGPT?

No, according to OpenAI. ChatGPT search relies on OAI-SearchBot; GPTBot is for training models.

Does blocking Google-Extended remove me from Google?

No. Google says it does not affect inclusion in Search or ranking. AI Overviews and AI Mode rely on Googlebot.

Do I need a special file to appear in AI?

No. Google says there are no additional requirements for its AI features. What matters is not blocking search crawlers.

Sources

More guides for hotels

More guides for hotels →

Español · Italiano · Português · Ελληνικά