Should you allow or block AI crawlers?
Start by deciding what you're optimizing for. There is no universally correct answer — it depends on whether you want AI visibility, content protection, or both.
AI crawlers fall into two rough categories:
- crawlers tied to live answer or citation products, which can send you referral traffic
- crawlers used purely for model training, which typically send no traffic back
Which AI crawlers actually exist
- GPTBot (OpenAI) — crawls content to train OpenAI's models
- OAI-SearchBot (OpenAI) — surfaces pages in ChatGPT's search/browsing features
- ClaudeBot (Anthropic) — crawls content for Anthropic's model training
- anthropic-ai — an older/alternate Anthropic user agent
- PerplexityBot — crawls and cites content in real-time answers
- Google-Extended — controls use of your content for Gemini and AI features
- Bingbot — feeds Copilot's AI answers in addition to Bing search
- Applebot-Extended — controls use of your content for Apple Intelligence
Why the audit-your-current-setup step comes first
Most site owners assume their robots.txt already handles this correctly. That is not true.
A quick check should confirm:
- whether a blanket disallow rule is silently blocking every AI bot
- whether a CDN or security plugin is blocking AI user agents outside robots.txt
- whether specific bots are already allowed or disallowed by name
- whether your rules match your actual intent
When blocking everything makes sense
Block all AI crawlers only when:
- your content is a core commercial asset you don't want used for training
- you run a paywalled or proprietary dataset site
- server load from aggressive crawlers is a real cost concern
- you have no interest in AI-driven referral traffic at this time
Most sites benefit from a more selective approach rather than blocking everything.
Sample robots.txt snippets
Allow everything:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Block everything:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Middle ground — block training bots, allow citation bots:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Allow: /
A simple decision framework
- Decide your stance: full access, full block, or training-vs-citation middle ground.
- Add the matching rules to your existing robots.txt file.
- Fetch your live robots.txt to confirm the rules saved correctly.
- Check your CDN or security plugin settings separately, since they can override robots.txt.
- Re-check every few months as new AI crawlers appear.
How this connects to your AI visibility
Being crawlable is step zero for AI visibility. No amount of schema markup or clean formatting matters if the crawler can't reach the page. If you've recently changed your robots.txt, confirm the fix by running an AEO/GEO score check.
Bottom line
A robots.txt file built only for Googlebot and Bingbot is now incomplete. Decide deliberately whether each AI crawler helps or costs you, then configure your rules to match — instead of leaving it to whatever the default happened to be.