If you publish anything worth reading, AI companies are probably already crawling it, and some of their crawlers are getting past the walls you built. Two stories this week show what’s at stake. One report says an Anthropic model got past a paywall to pull Georgia vote-count data. Elvis Presley’s rights holders are suing over 274 songs they say were used for training. A block AI crawlers robots.txt rule takes five minutes to add and is the bare minimum, but on its own it enforces nothing. This guide gives you robots.txt directives, Cloudflare rules and server-level blocks you can deploy today, and shows which ones actually stop a determined scraper.
What’s new: why “block AI crawlers robots.txt” is trending again
Neither story involves a new technique. They matter because they hit nerves publishers already had. The Georgia report describes a model retrieving data that sat behind a paywall. It may have used a live browsing agent, a cached copy, or text that leaked into the page source. Either way the lesson holds: content that looks gated to a human often isn’t gated to a machine. Many paywalls send the full article to the browser and then hide it with JavaScript. A crawler that reads raw HTML never sees the overlay.
The Presley lawsuit is the other half of the problem. Rights holders are targeting training itself, not just output that copies their work. Each new lawsuit makes a clear, timestamped opt-out record more valuable. A robots.txt disallow for GPTBot or CCBot is not a legal shield, but it shows clear intent. That will matter as courts and regulators decide what “consent” means for training data.
The tooling has changed too. Cloudflare now blocks known AI crawlers by default on new domains, offers a managed robots.txt, and provides an AI Crawl Control dashboard showing which bots hit your site and whether they follow your rules. Most major AI vendors now run separate user agents for training, search indexing and user-triggered fetches. That gives you finer control, but only if your rules name the right agents.
Why it matters
- Robots.txt is a request, not a lock. Reputable crawlers such as GPTBot, ClaudeBot and CCBot say they honor it. Others, including Bytespider in the past and many unidentified scrapers, have been caught ignoring it. You need enforcement at the edge or the server.
- Training and search are now separate decisions. Blocking GPTBot keeps your content out of OpenAI training, but OAI-SearchBot and ChatGPT-User handle ChatGPT search and live fetches. Block everything and you may vanish from AI answers that send you referral traffic.
- The Google-Extended opt-out does not affect Google Search. Google-Extended is a robots.txt control token, not a separate crawler. Googlebot does the fetching, so you cannot block Google-Extended at the firewall without also blocking Search.
- Browser-side paywalls leak. If the full article sits in the HTML and JavaScript hides it, any bot can read it. To stop AI scraping of paywalled content, the server must withhold the text.
- Your opt-out record has legal and commercial value. Licensing deals and pay-per-crawl programs assume you can prove control over access. A documented block strengthens your position in both.
- User agents can be faked. Anyone can send “GPTBot” in a header. Good enforcement checks published IP ranges or uses a verified-bot list such as Cloudflare’s.
How to block AI crawlers today
1. Decide your policy before writing rules
Most business sites should block training crawlers and allow AI search and user-fetch agents, so they keep appearing, with citations, in ChatGPT, Perplexity and Claude answers. Publishers whose main asset is paywalled content may want to block everything. Write the decision down and check every rule below against it.
2. Add AI crawler rules to robots.txt
Put this at https://yourdomain.com/robots.txt. It blocks the major training crawlers and leaves regular search engines alone:
# --- AI training crawlers: blocked ---
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
# --- AI search / answer engines: choose one ---
# Option A: allow (keeps citations and referral traffic)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Option B: block (replace the two blocks above)
# User-agent: OAI-SearchBot
# Disallow: /
# User-agent: PerplexityBot
# Disallow: /
# --- Everyone else ---
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpTo protect only paywalled sections, replace Disallow: / with specific paths such as Disallow: /premium/. On WordPress, check whether an SEO plugin generates a virtual robots.txt. If it does, add the rules in the plugin’s editor, because a physical file can conflict with it.
3. Turn on Cloudflare AI bot blocking
If your site runs through Cloudflare, go to Security > Settings (or Security > Bots on older dashboards) and enable Block AI bots. Then open AI Crawl Control to see which crawlers hit your site and set allow or block for each. Cloudflare’s managed robots.txt adds standard AI opt-out directives to your existing file without overwriting it.
4. Add a Cloudflare WAF custom rule for precise control
The one-click toggle is broad. A custom rule lets you pick the agents. Go to Security > WAF > Custom rules > Create rule, select Edit expression, and paste:
(http.user_agent contains "GPTBot")
or (http.user_agent contains "ClaudeBot")
or (http.user_agent contains "CCBot")
or (http.user_agent contains "Bytespider")
or (http.user_agent contains "meta-externalagent")
or (http.user_agent contains "Amazonbot")
or (http.user_agent contains "PerplexityBot")Set the action to Block. To catch every crawler Cloudflare has verified as an AI crawler, including ones you didn’t name, use the verified-bot category field:
(cf.verified_bot_category eq "AI Crawler")To guard only paywalled paths, append and starts_with(http.request.uri.path, "/premium/") to either expression.
5. Block at the server if you’re not on Cloudflare
For Nginx, add a map in the http block and a check in each server block:
map $http_user_agent $ai_bot {
default 0;
~*(GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot|Amazonbot|meta-externalagent|Applebot-Extended) 1;
}
server {
# ...existing config...
if ($ai_bot) {
return 403;
}
}For Apache or LiteSpeed, common on shared WordPress hosting, add this near the top of .htaccess:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot|Amazonbot|meta-externalagent) [NC]
RewriteRule .* - [F,L]
</IfModule>Leave Google-Extended out of both lists. It never appears as a user agent, so the rule would do nothing.
6. Make the server enforce your paywall
Non-subscribers should receive only a teaser in the HTML, with the full text loaded after authentication. To check, run this against a locked article:
curl -s -A "GPTBot" https://yourdomain.com/premium/sample-article | grep -c "a sentence from paragraph 6"Any result above zero means the server sends the full text to anyone who asks, and every crawler can read it. If you use Google’s flexible sampling, mark gated sections with isAccessibleForFree: false structured data so Googlebot doesn’t treat the setup as cloaking.
7. Verify, then monitor
Test the block:
curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com/You should get a 403. Then review your access logs or Cloudflare analytics weekly. Look for crawlers that ignore robots.txt, and for browser user agents requesting hundreds of pages per minute from datacenter IP ranges. That pattern usually means a stealth scraper.
How the blocking methods compare
| Method | Setup time | Enforces the block? | Stops spoofed agents? | Best for |
|---|---|---|---|---|
| robots.txt directives | 5 minutes | No, crawlers comply voluntarily | No | Every site; also a documented opt-out record |
| Cloudflare “Block AI bots” toggle | 1 minute | Yes, at the edge | Partly, through bot fingerprinting | Sites already on Cloudflare that want a broad block |
| Cloudflare WAF custom rule | 10 minutes | Yes, at the edge | Yes, with the verified-bot fields | Blocking specific bots or protecting paywalled paths |
| Nginx / Apache user-agent block | 10-20 minutes | Yes, at the server | No, matches the header only | Sites not behind a CDN |
| Server-side paywall | Varies (developer work) | Yes, the content is never sent | Yes | Publishers whose revenue depends on gated content |
Our take: layer them. Robots.txt documents your intent, Cloudflare or the server enforces it, and a server-side paywall protects the content that pays your bills.
What’s next
The line between crawler and user will keep blurring. AI agents that browse on a user’s behalf, such as ChatGPT agent mode, Claude’s browsing and Perplexity Comet, often run a real browser and look much like human visitors. Vendors argue these fetches are user actions rather than crawling, and some say robots.txt doesn’t apply. That fight is far from settled, and the Georgia paywall report adds pressure for a clear standard on what agents may access behind a login or paywall.
Commercially, Cloudflare’s pay-per-crawl and similar licensing marketplaces are growing. Content valuable enough to block may be valuable enough to license, and blocking first gives you leverage. Also track the IETF’s AI preferences work, which aims to replace today’s patchwork of vendor-specific tokens with a standard vocabulary for training, search and inference permissions.
Legally, the Presley case and other pending training-data lawsuits will shape how much weight courts give opt-out signals. Keep dated copies of your robots.txt and firewall configurations. If your content ever becomes evidence, that record strengthens your case.
Frequently Asked Questions
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI’s crawler and has nothing to do with Google Search. Even the Google-Extended opt-out leaves Search rankings untouched. It only controls whether Google uses your content to train and ground Gemini models.
What is the PerplexityBot user agent, and does blocking it remove me from Perplexity?
PerplexityBot is the crawler Perplexity uses to index pages for its answer engine. Blocking it in robots.txt or at the firewall makes citations of your pages much less likely. Perplexity-User handles real-time fetches when someone asks about a specific URL and follows different rules, so block it separately if you want out entirely.
Does a CCBot Common Crawl block remove content that’s already been collected?
No. A block only stops future crawls. Earlier Common Crawl snapshots remain in public archives and may already sit in training datasets. Blocking now still keeps new and updated content out of future snapshots.
Is robots.txt enough to stop AI scraping of paywall content?
No. Robots.txt is voluntary, and a paywall that hides text with JavaScript still sends the full article in the HTML. Have the server withhold content from unauthenticated requests, and add Cloudflare or server-level blocks for crawlers that ignore your rules.
Should a small business block AI crawlers at all?
Block the training crawlers and allow the AI search agents. Your content stays out of model training, and you stay visible when customers ask ChatGPT or Perplexity for recommendations. Most small business sites gain more from AI citations than they lose to scraping.
How do I know if a bot is faking its user agent?
Compare the request IP with the vendor’s published ranges; OpenAI, Anthropic, Google and others list theirs. On Cloudflare, rely on verified-bot detection rather than user-agent strings. High-volume, bot-like browsing from cloud-hosting IPs with a browser user agent is the most common sign of a disguised scraper.
Go deeper than this article
This article covers the essentials. Our premium eguide library gives you the full step-by-step playbooks — prompts, workflows, and copy-paste recipes you can put to work today.
