Blocking AI Crawlers: Find Out What Your Host Switched On

October 11, 2026

An office manager circles a line on a printed page under a desk lamp at dusk. (AI-generated image)

Key takeaways

Possibly, and nobody may have asked you.
Hosts and network providers such as Cloudflare can add robots.txt rules that block AI training crawlers while Google keeps ranking you, and a broader setting can also shut out the AI search bots patients rely on.

Possibly, and nobody may have asked you. Hosts and network providers such as Cloudflare can add robots.txt rules that block AI training crawlers while Google keeps ranking you, and a broader setting can also shut out the AI search bots patients rely on. Ask your developer what is blocked, by which rule and who changed it, then decide for yourself.

What does blocking AI crawlers actually change on a practice website?

Blocking AI crawlers adds lines to your robots.txt file, or a firewall rule, that ask named AI bots to stay out. Your pages stay live, and Googlebot can still visit unless a rule names it too.

The robots.txt file sits at the root of your site and gives each web crawler its instructions. Cloudflare’s documentation says its managed setting generates and maintains a robots.txt file that tells known AI crawlers to stay away from your content. The same page warns that compliance is voluntary: the file states a preference rather than locking a door.

Enforcement is a separate layer. Cloudflare points owners who want a hard block to AI Crawl Control, which refuses requests at the network. Either way, the change happens above your pages, where nobody in the office looks.

Can your host change your robots.txt without telling you?

Yes. A host or network provider can add rules to the file that bots see, while your own copy never changes.

With the managed setting on, Cloudflare will prepend its managed content before your original rules. The audit tool MetricSpot describes that managed block disallowing GPTBot or ClaudeBot even when the owner’s own file allows them. Defaults matter too: hosting.com reports that new domains and free-tier sites get training and agent crawlers blocked on pages that display ads.

Search Engine Journal’s objection is fair: the setting “hands a vendor the decision about what your website tells AI crawlers,” its analysis argues. The real problem is not the rule. It is the missing signature.

Is blocking AI crawlers hiding your practice from patients?

It depends on which crawler you block. Refusing a training crawler protects your writing; refusing a search crawler removes you from the answers patients get.

The big AI companies now split those jobs. OpenAI says a site can allow OAI-SearchBot to appear in its results while disallowing GPTBot, using separate robots.txt tags for each. Anthropic lists three separate bots for training, user requests and search, each controlled on its own.

Some lines are blunter. Common Crawl notes that blocking CCBot keeps your pages from those who use the data for search indexing as well as from model training. So the test is simple. Block crawlers that only train, and you are protecting. Block the search or user agents, and you are hiding. For what answer engines need to find once they get in, see what practices must publish.

What three questions should you ask your developer or host?

Get the answers in writing, because the next person to touch the site will need them.

What is blocked? Open yoursite.com/robots.txt in a browser and check every user-agent line. Ask for a plain list of each crawler refused, and whether it trains models, powers answers or fetches a page when a person asks.
By which rule? A refusal can live in your own file or in a managed block above it. It can also sit in a firewall setting or a plugin’s code. Fixing the wrong layer changes nothing.
Who changed it, and when? Ask for a name and a date. If the answer is a default nobody chose, that is your finding.

Agencies can make this part of every client handover. A practice that knows its own settings can’t be surprised by them.

Are there good reasons to refuse training crawlers?

Yes. Your articles are your work, and you may not want them feeding large language models that never send a patient back.

There are practical reasons too. Common Crawl says it knows of crawlers falsely identifying themselves as CCBot. A bot’s name is not proof of who sent it. Bandwidth is another; its FAQ shows how a Crawl-delay slows the bot rather than refusing it access.

My view of any program, a body or a website, is that a change without a name next to it is not a decision. Refusing training crawlers can be the right call. It should be your call.

How do you keep the decision yours once it is made?

Write the policy down, own the file, and recheck it after every host, plugin or theme change.

Cloudflare’s sync sets policy by category, not by crawler. Search Engine Journal reports that the fix for finer control is to turn the sync off and maintain the file yourself. That suits a practice with a developer on call. A short note with the date and the reason keeps the next agency from undoing it.

Then give the bots you allow something worth finding. Silence costs more than any bot setting, as the post on competitor content you never published shows. The first app I built, Blog & Social Posts, Done for You, has run on my own four sites since September 2026.

Frequently asked questions

Will blocking AI crawlers remove my site from existing data sets?

No. A robots.txt rule speaks to future visits and gives no instruction about copies a crawler already collected. If removal from an existing data set matters to you, contact the company that runs the crawler and ask what its process is. Treat older copies as a separate request from the block itself.

Will blocking AI crawlers stop Google from indexing my site?

Not if the rules name only AI bots. A block written for GPTBot or ClaudeBot leaves Googlebot free to crawl and rank your pages. The danger is a broad rule under User-agent: * with Disallow: /. That tells every crawler to stay out, Google and other search engines included. Look for that line first.

How often should a practice check its robots.txt file?

Check it after any change to your host, network provider, security plugin or theme, and on a regular schedule in between. Each of those can rewrite what bots see without touching your pages. Save a copy each time, so you can compare versions and spot which line moved.

Can I let AI answer engines into some pages but not others?

Yes. Each user-agent group in robots.txt can carry its own allow and disallow lines for specific folders. For example, a practice might open its service pages and articles while refusing a folder of downloadable forms. Keep the rules short, because every extra line is another place for a conflict to hide.

Should an agency decide AI crawler settings for its clients?

An agency should recommend, and the practice owner should decide. Explain the training-versus-search trade-off in plain words, record the owner’s choice, and file it with the handover notes. That record protects the agency when a host default later shifts, and it gives the next developer a reason instead of a mystery.

The Bottom Line

Treat AI crawler access as a choice with your name on it. Keep training bots out if you like. Keep the answer engines in if you want patients to find you there. Make your developer or host confirm the result in writing.

Then publish what those engines should find. Try it: start with Blog & Social Posts, Done for You, compare the plans, or begin with the free book.

Start free for 14 days

Start free for 14 days

Sources

Jim Goetz, Founder of PublishSense