The robots.txt you did not write
Cloudflare rewrites robots.txt at the edge and blocks AI crawlers by default. None of it lives in your repository, so reading the file in git tells you nothing.
I was auditing my own site for AI search visibility. Part of that is boring: check that nothing is telling crawlers to go away. I opened public/robots.txt in the repository, saw four permissive lines, and nearly moved on.
Then I fetched the live file instead. It was not the same file.
What was actually being served
The version on the edge carried a block of rules that exist nowhere in the project:
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /The full list ran to nine crawlers: ClaudeBot, GPTBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent and CloudflareBrowserRenderingCrawler. There was also a Content-Signal header declaring search=yes,ai-train=no,use=reference.
I had not written any of it. Cloudflare had, and it was on by default.
Why this one is easy to miss
Every instinct says robots.txt is a static file. It is in the repository, it is in the deploy, and if you want to know what it says you open it. That instinct is exactly what hides this.
The rewrite happens at the edge, after your origin has already returned the correct file. Your build is fine. Your deploy is fine. Nothing errors, nothing warns, and no log line anywhere mentions it. The only way to see it is to ask the internet rather than your editor.
The two switches
Both live on the same zone endpoint, GET/PUT /zones/{zone}/bot_management:
ai_bots_protection: "block"turns on the AI bot blocking and the Disallow rules that advertise it.is_robots_txt_managed: truelets Cloudflare rewrite robots.txt at the edge.
Turning off the first one is not enough, which cost me a round trip. With ai_bots_protection disabled the served file was unchanged. Both settings have to go off, and then the cached copy has to be purged:
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
--data '{"files":["https://example.com/robots.txt"]}'With is_robots_txt_managed set to false, the file from your repository is served verbatim again. That is the state worth keeping. Robots policy belongs in git, where it can be reviewed, not in a dashboard toggle somebody flipped once.
This is a real choice, not just a bug
Blocking AI crawlers is a legitimate position. If your content is the product, you may not want it in a training set, and Cloudflare turning it on by default is defensible.
The problem is that it is silently incompatible with the opposite goal. If you want to be the answer when somebody asks ChatGPT or Claude or Perplexity about your niche, you have to be readable. Those two goals are in direct conflict, and the default picks one for you without telling you.
That is fine when it is a decision. It is not fine when nobody in the team knows it happened.
The two minute check
Fetch your own robots.txt and compare it to the one in your repository:
curl -s https://your-domain.com/robots.txtIf the live file has rules your repository does not, something in front of your origin is writing them. On Cloudflare it is almost certainly these two settings. Check it on every site where being found by AI search matters, because nothing in the codebase will ever reveal it.
On this site the two files match again. That took a settings change and a cache purge, and no amount of reading the repository would have told me it was needed.