Giving AI Bots Access to Your Website: GPTBot, ClaudeBot and Google-Extended Explained
More and more guests aren’t asking Google for a restaurant tip, they’re asking ChatGPT or Perplexity. Handy, since a good AI assistant hands back a short, concrete shortlist within seconds. Less handy if your restaurant never makes that list, even though your website is otherwise in great shape. One overlooked reason: the AI bot that needs to read your site never gets in.
What are AI bots?
Alongside familiar search engine bots like Googlebot, AI companies run their own crawlers across the web to read websites. These AI bots gather the content an assistant uses to build an answer, or that trains a model in the first place. The main ones you’ll come across:
- GPTBot – OpenAI’s crawler, used for ChatGPT.
- ClaudeBot – Anthropic’s crawler, used for Claude.
- PerplexityBot – used by Perplexity to build up-to-date answers.
- Google-Extended – separate from the regular Googlebot, this one decides whether Gemini and Google’s AI Overviews can use your content.
- Applebot-Extended – for Apple Intelligence.
- CCBot – Common Crawl’s crawler, a data source many AI models are partly trained on.
Each bot gets its own rule in your robots.txt file. If that rule is missing, or if it carries a Disallow, that specific bot isn’t allowed to read your pages.
Why this decides whether AI can recommend you
An AI assistant can only recommend what it can actually read. Block GPTBot, and ChatGPT simply can’t cite your restaurant, no matter how good your menu or your service is. That goes straight to the heart of your AI visibility: how often your restaurant turns up in AI answers and AI summaries.
The sting is in the word “accidentally”. Hardly any restaurant owner blocks an AI bot on purpose. It usually happens through a security plugin, a hosting package with a broad default firewall rule, or a robots.txt file that was set up once and never checked again.
How your site can end up blocking AI bots by accident
A robots.txt file is harmless enough on its own: it neatly sets out which bots may visit which parts of your site. But some security settings add broad rules on top, like “block all unknown bots” or “block anything that looks like an AI crawler.” Meant to keep out scrapers and spam, but a rule like that often catches exactly the AI bots you wanted to let in.
A firewall at your hosting provider, or a separate security package, can also block bots by user-agent, without that showing up anywhere in your own site settings.
How to check which AI bots have access
- Open
yourdomain.com/robots.txtin your browser. - Look for rules starting with
User-agent: GPTBot,ClaudeBot,PerplexityBot,Google-ExtendedorApplebot-Extended. - If one of those user-agents has
Disallow: /next to it, that bot is fully blocked. - If you don’t see those user-agents at all, the general rule at the bottom of the file usually applies. That’s often permission, but not always, so check it.
- Not sure? Test the file with a free online robots.txt checker and enter the bot’s name.
- Still stuck? Ask your hosting provider or platform whether a firewall rule for AI bots is active outside of your robots.txt file.
When blocking a bot actually makes sense
Not every bot needs to be let in automatically. Some restaurant owners deliberately don’t want their content used to train AI models, regardless of whether they want to appear in an answer. That distinction exists: you can allow bots that fetch live answers while blocking training crawlers like CCBot separately. If you’re unsure what fits your business, choose per bot rather than blocking or allowing everything at once.
How this fits into your bigger AI visibility picture
Access for AI bots is the foundation: without it, they never read your site, however good your content is. Only after that do the other building blocks count, like an llms.txt file that points AI assistants to your most important pages, and structured data about your cuisine, location and menu. Want one overview of exactly where your site stands, bot access included? Check your AI Visibility Score.
Want to make sure AI bots are actually welcome on your site? Try Sitebite for free and let the technical basics, from your sitemap to bot access, take care of themselves.
