Website owners today face a new challenge: AI companies are crawling their sites to train models, but unlike search engines that send traffic back, these AI crawlers offer little in return. While Google sends one visitor for every 14 times it crawls your site, OpenAI crawls 1,700 times for each visitor it sends back, and Anthropic’s ratio is even worse at 73,000 to 1.
Cloudflare has introduced two simple tools to help website owners take control. You can now automatically block AI training crawlers while keeping your site visible to search engines, and you can choose to block AI bots only on pages where you show ads.
How to Set Up AI Bot Protection
Method 1: Enable Managed Robots.txt
Step 1: Log into your Cloudflare dashboard
Step 2: Navigate to your website settings
Step 3: Look for the “Managed robots.txt” option and toggle it on
Step 4: Cloudflare will automatically create or update your robots.txt file
This method tells AI crawlers like GPTBot, ClaudeBot, and others not to use your content for training. It keeps your site visible to Google and other search engines for normal search results. Cloudflare updates the file automatically as new AI bots appear.
(Image courtesy of Cloudflare)
Method 2: Block AI Bots Only on Pages with Ads
Step 1: Go to Security > Settings > Bots in your Cloudflare dashboard
Step 2: Choose “Block on pages with Ads” instead of “Block Everywhere”
Step 3: Cloudflare will automatically detect which pages show ads and block AI bots only on those pages
This option is perfect if you want AI bots to access some content (like help documentation) but protect the pages where you earn money from advertising.
(Image courtesy of Cloudflare)
Frequently Asked Questions
Will this affect my search engine rankings?
No, these tools specifically target AI training bots while allowing search engines like Google, Bing, and others to continue indexing your site normally.
How does Cloudflare detect which pages have ads?
Cloudflare scans your web pages for common advertising code patterns, links to ad servers like Google AdSense, and uses Content Security Policy reports to identify ad-serving pages.
What if I already have a robots.txt file?
Cloudflare will add its AI bot blocking rules to the top of your existing file, so your current settings remain unchanged.
Can AI bots ignore robots.txt files?
Yes, robots.txt is voluntary. However, Cloudflare also offers active blocking that physically prevents bots from accessing your content, regardless of whether they respect robots.txt.
Does this cost extra?
No, both features are available to all Cloudflare customers, including those on the free plan.
How often does Cloudflare update the managed robots.txt?
Cloudflare automatically updates the file as new AI bots emerge, so you don’t need to monitor or maintain it yourself.
Will this block legitimate AI tools that my visitors use?
The blocking targets crawlers that scrape your site for training data, not AI tools that your actual visitors might use in their browsers.
Conclusion
The relationship between websites and crawlers is changing. While search engines have traditionally provided value by sending traffic back to sites they crawl, AI training bots extract content without offering much in return.
Cloudflare’s new tools give website owners an easy way to protect their content without sacrificing search engine visibility or requiring technical expertise. Whether you choose the managed robots.txt approach or prefer to block AI bots only on monetized pages, you can now control how your content is used for AI training.
With over one million websites already using these protections, website owners are taking back control of their content. The choice of how your work gets used should be yours to make.
