Block AI training, stay visible in search: what Cloudflare’s new setting means for your website

Reading time: 5 minutes · For owners of SMBs in Ausserschwyz, the Linthgebiet and around Lake Zurich whose website should bring in enquiries

An SMB’s website is rarely looked after by anyone full time. It runs at the hosting provider, a login is stored somewhere, and when a setting needs changing, it’s done by whoever does it “on top of everything else”. That is exactly where a decision landed on 15 September that hardly anyone has made consciously so far: may AI use your content for training, and what happens to your visibility if you say no?

Cloudflare, the service that sits in front of a great many websites as a layer of protection and speed, has introduced a new setting for this. I found the announcement as a newsletter in my inbox and put it into context on camera:

Why it used to be either-or

Cloudflare is something like the bouncer between your website and its visitors. Around a fifth of the web runs through the company’s network. It delivers pages faster, fends off attacks and decides which bots get in. When this bouncer changes the house rules, they change for a large part of the internet.

The problem until now: “block AI bots” was a sledgehammer. Some of the biggest providers use the same crawler for search and for training their models. Locking out one meant locking out the other. Accordingly, Cloudflare’s announcement says fewer than 1 per cent of websites blocked search, but 17 per cent set up some form of training block. Most businesses want to be found. But many don’t want their work to end up in the next AI model without being asked.

Three switches instead of one

Since the update, you decide separately on three things: search, AI training and agents, meaning assistants that visit a page on behalf of a person. The new “Disallow AI Training” setting writes your preference into robots.txt and blocks the pure training crawlers, such as those from Amazon, Anthropic, Meta and OpenAI. The search crawlers from Google, Bing and Apple can keep indexing. According to Cloudflare, Google and Apple have stated that opting out of training does not affect search rankings.

The point I stress in the video: a training block doesn’t mean ChatGPT or Claude can no longer open your page when someone asks about it. That is handled by the agents switch, and it is separate. You can stay present in AI answers and still say: my content is not used for training. From a data security point of view, this is for me one of the bigger updates of recent weeks, perhaps months: businesses get much more control over what large AI providers may do with their content.

What to do now

1. Find out whether your website runs behind Cloudflare at all

Many websites in the region run directly at the hosting provider, without Cloudflare in between. Ask your web agency or check yourself who manages your domain. If it runs through Cloudflare, you’ll find the new switches in the domain’s security settings. According to Cloudflare, they are available on all plans, including the free one.

2. Don’t click “Block” if you want to be found

This is the trap. Since the update, “Block” means something different from before: it now also locks out Googlebot, Bingbot and Applebot, so search goes with it. If you only want to stop training, choose “Disallow AI Training”. According to Cloudflare, existing settings are carried over automatically. Still, check once what is set for your site. A wrong tick here doesn’t cost you clicks, it costs you your visibility in the region.

3. Decide which content is your edge

The real question isn’t a technical one. If your website mostly repeats what is written everywhere anyway, a training block protects little. If it contains real expertise, such as costing aids, process descriptions or experience from hundreds of jobs, then you get to decide whether that becomes free training material. Opening hours, services and references should be found. Your expertise should above all convince customers, as I described in the post on pain points instead of keywords.

4. Don’t lock out agents by reflex

An agent that visits your page does so on behalf of a person who has a question right now: who does natural stone work in Lachen, who can deliver next week? If you block agents, you are missing from exactly that answer. My starting point for most SMBs: search on, training a conscious decision, agents not blocked by reflex. How to stay visible in search engines and AI assistants is one of the building blocks of local visibility.

In many businesses, settings like these get left undone because nobody is responsible for them. That is exactly what marketing without a marketing department is for: one contact person who keeps an eye on the website, visibility and fundamental questions like this.

Where the limits are

If your website doesn’t run through Cloudflare, your main tool is robots.txt. There you can, for example, exclude Google-Extended or Applebot-Extended from training. But that is a request, not a block: a robots.txt can’t check who is crawling and can’t stop a crawler that ignores it. Even with Cloudflare, not everything is solved. According to Cloudflare, Bing will only adopt the training preference from robots.txt from early 2027, and the question of how much of your content appears in AI summaries is something Cloudflare only plans to tackle next. The training block is a good first step, but not complete protection.

Frequently asked questions

Will I disappear from Google if I block AI training?

Not if you choose “Disallow AI Training” in Cloudflare. That setting still lets in the search crawlers from Google, Bing and Apple. The “Block” setting, on the other hand, has also locked out search since 15 September.

Can ChatGPT or Claude still open my website afterwards?

Yes. Whether assistants may open your page on behalf of a person is controlled by the separate agents switch. The training block only affects training AI models on your content.

Does the new setting cost anything?

According to Cloudflare, the new controls are available on all plans, including the free one. The requirement is that your domain runs through Cloudflare.

Should I block AI training on my website?

That depends on what is on your website. For general information, a block protects little. For your own expertise that gives you your edge, a conscious decision is worthwhile. For most SMBs, search and agents should stay open.