# Cloudflare Now Blocks AI Crawlers by Default on Ad-Supported Pages

Page: https://hirantha.me/blog/cloudflare-now-blocks-ai-crawlers-by-default-on-ad-supported-pages
Published: 2026-09-16
Written by: hirantha.me Newsroom (AI-generated). An AI model wrote this from the sources listed below; how it is made: https://hirantha.me/about/newsroom
Topics: cloudflare, ai crawlers, robots.txt, web publishing, bot management

> As of September 15, Cloudflare's default settings block AI training and agent crawlers on any page carrying ads, while search bots stay allowed.

## What changed on September 15

Cloudflare's default bot-handling rules changed for a large slice of the web this week. As of September 15, 2026, any page that carries advertising and sits on an affected Cloudflare zone will, by default, allow search crawlers to keep indexing it while blocking crawlers that exist to feed AI training or AI agents. Cloudflare is a content delivery and security service that sits in front of a large share of the web, which is why a default change to its bot rules has an effect well beyond Cloudflare's own customers.

The new defaults sort bot traffic into categories, including search, agent, training, transactional, data collection, security testing, search engine optimization (SEO), ads verification, social and link preview, feed fetching, and monitoring. Of these, only training and agent crawlers are blocked by default on ad-supported pages; search crawlers are allowed everywhere under the new rules.

## Who this actually applies to

The change is not retroactive for everyone. It applies to domains onboarding to Cloudflare from September 15 onward, new sites created by existing customers, and zones on Cloudflare's free tier. Existing paying customers keep whatever bot settings they already had unless they go in and change them. That scoping matters: a publisher who has been on a paid Cloudflare plan for years will not see their crawler access change automatically, while anyone setting up a new site on Cloudflare today inherits the new, more restrictive defaults from day one.

## The mixed-use crawler problem

The hardest case is bots that serve more than one purpose from a single crawl, sometimes called mixed-use crawlers, since a general-purpose bot like Googlebot can feed both a search index and an AI system from the same pass over a page. Cloudflare's rule treats a mixed-use crawler according to its most-restricted purpose. In practice, that means if a bot's traffic includes any AI training use, it gets blocked wherever training is blocked, even if the same bot also does legitimate search indexing. A crawler operator that wants search access preserved under the new defaults has to be able to separate its search traffic from its training and agent traffic, rather than sending both through one undifferentiated crawler.

Cloudflare paired the policy with an extension to the `robots.txt` standard it calls Content-Signal, which lets a site declare, per purpose, whether crawling is allowed: for search indexing, for reference or citation-style AI use, or for full AI training use. It also renamed its verified-bot program's meaning in the process: being a verified bot no longer implies blanket permission to crawl, only that the bot is allowed within whatever category it has declared and been verified for.

## The money angle

Alongside the blocking defaults, Cloudflare is evolving its existing Pay Per Crawl marketplace into what it calls Pay Per Use, aimed at compensating publishers not just when a crawler visits but when their content generates downstream value, such as appearing in an AI-generated search answer or being accessed through a premium content tier. Ceramic.ai and You.com are named as launch partners for that model, giving publishers on those platforms a way to get paid when their content is the thing surfacing an answer, rather than relying on ad impressions from a human visitor who may never arrive because the AI answer already satisfied them.

## Where the disagreement is

Google has pushed back on how the change is being framed, pointing to its own Google-Extended token, an existing opt-out mechanism that lets a site block its content from Google's AI training uses without affecting how that same site ranks in ordinary Google search. Google's argument is that publishers already had a way to separate search visibility from AI training consent before Cloudflare's default change, and that Cloudflare blocking mixed-use bots by default overstates how entangled the two uses actually are in practice.

## What it means in practice

For anyone running a new site or a free-tier zone on Cloudflare, ad-supported pages will now surface in search as before while being closed to AI training and agent crawlers unless a site owner explicitly reopens them, which is worth checking in the Cloudflare dashboard against a site's actual monetization strategy. For teams that publish content and want AI visibility, such as being cited in AI-generated answers, the practical step is adopting the Content-Signal `robots.txt` directives explicitly rather than relying on default behavior, since the whole point of the new system is that a blanket "verified bot" allowance no longer guarantees crawl access to every use a bot might have.

## Questions

**Does Cloudflare's change block AI crawlers on my site automatically?**
Only if your site is a new domain onboarding to Cloudflare from September 15, 2026 onward, a new site from an existing customer, or on Cloudflare's free tier. Existing paid customers keep their current settings unless they change them.

**Will this hurt my site's search rankings?**
No. Search crawlers are allowed on all pages under the new defaults; only AI training and AI agent crawlers are blocked on ad-supported pages by default.

**What happens to bots like Googlebot that serve more than one purpose?**
Cloudflare treats a mixed-use crawler according to its most-restricted purpose, so if any of its traffic is used for AI training, it gets blocked wherever training crawlers are blocked, even on the same pass that also does search indexing.

**How can publishers control this more precisely?**
Cloudflare introduced a Content-Signal extension to robots.txt that lets a site declare separate permissions for search indexing, AI reference use, and full AI training use, rather than relying on a single blanket allow or block.

**Is there a way for publishers to get paid for AI use of their content?**
Cloudflare is expanding its Pay Per Crawl marketplace into a Pay Per Use model, with Ceramic.ai and You.com as launch partners, aimed at compensating publishers when their content drives value in AI search answers or premium access, not just when a crawler visits.

## Sources

- [Cloudflare's September 15 AI Crawler Wall: What Actually Changes](https://www.novaproxy.io/blog/cloudflare-will-block-ai-crawlers-by-default-on-september-15-2026-heres-what-actually-changes), novaproxy.io
- [Cloudflare's new policy pushes AI companies to pay for publishers' content](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/), TechCrunch