[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-post-robots-txt-you-did-not-write":3},{"name":4,"created_at":5,"published_at":6,"updated_at":7,"id":8,"uuid":9,"content":10,"slug":185,"full_slug":186,"sort_by_date":187,"position":188,"tag_list":189,"is_startpage":190,"parent_id":191,"meta_data":187,"group_id":192,"first_published_at":193,"release_id":187,"lang":194,"path":187,"alternates":195,"default_full_slug":187,"translated_slugs":187},"The robots.txt you did not write","2026-09-03T12:39:48.565Z","2026-09-03T12:59:33.165Z","2026-09-03T12:59:33.178Z",216062314778581,"d3d25fc7-3b63-43db-b05d-a1e9fe919f67",{"_uid":11,"body":12,"title":4,"excerpt":182,"component":183,"publish_at":184},"d23817ac-a382-4789-bd92-97c198c3e1a1",{"type":13,"content":14},"doc",[15,28,32,39,43,48,64,68,73,77,81,86,96,120,130,134,144,149,153,157,161,166,170,174,178],{"type":16,"content":17},"paragraph",[18,21,26],{"text":19,"type":20},"I was auditing my own site for AI search visibility. Part of that is boring: check that nothing is telling crawlers to go away. I opened ","text",{"text":22,"type":20,"marks":23},"public\u002Frobots.txt",[24],{"type":25},"code",{"text":27,"type":20}," in the repository, saw four permissive lines, and nearly moved on.",{"type":16,"content":29},[30],{"text":31,"type":20},"Then I fetched the live file instead. It was not the same file.",{"type":33,"attrs":34,"content":36},"heading",{"level":35},2,[37],{"text":38,"type":20},"What was actually being served",{"type":16,"content":40},[41],{"text":42,"type":20},"The version on the edge carried a block of rules that exist nowhere in the project:",{"type":44,"content":45},"code_block",[46],{"text":47,"type":20},"User-agent: ClaudeBot\nDisallow: \u002F\n\nUser-agent: GPTBot\nDisallow: \u002F\n\nUser-agent: Google-Extended\nDisallow: \u002F\n\nUser-agent: CCBot\nDisallow: \u002F",{"type":16,"content":49},[50,52,56,58,62],{"text":51,"type":20},"The full list ran to nine crawlers: ClaudeBot, GPTBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent and CloudflareBrowserRenderingCrawler. There was also a ",{"text":53,"type":20,"marks":54},"Content-Signal",[55],{"type":25},{"text":57,"type":20}," header declaring ",{"text":59,"type":20,"marks":60},"search=yes,ai-train=no,use=reference",[61],{"type":25},{"text":63,"type":20},".",{"type":16,"content":65},[66],{"text":67,"type":20},"I had not written any of it. Cloudflare had, and it was on by default.",{"type":33,"attrs":69,"content":70},{"level":35},[71],{"text":72,"type":20},"Why this one is easy to miss",{"type":16,"content":74},[75],{"text":76,"type":20},"Every instinct says robots.txt is a static file. It is in the repository, it is in the deploy, and if you want to know what it says you open it. That instinct is exactly what hides this.",{"type":16,"content":78},[79],{"text":80,"type":20},"The rewrite happens at the edge, after your origin has already returned the correct file. Your build is fine. Your deploy is fine. Nothing errors, nothing warns, and no log line anywhere mentions it. The only way to see it is to ask the internet rather than your editor.",{"type":33,"attrs":82,"content":83},{"level":35},[84],{"text":85,"type":20},"The two switches",{"type":16,"content":87},[88,90,94],{"text":89,"type":20},"Both live on the same zone endpoint, ",{"text":91,"type":20,"marks":92},"GET\u002FPUT \u002Fzones\u002F{zone}\u002Fbot_management",[93],{"type":25},{"text":95,"type":20},":",{"type":97,"content":98},"bullet_list",[99,110],{"type":100,"content":101},"list_item",[102],{"type":16,"content":103},[104,108],{"text":105,"type":20,"marks":106},"ai_bots_protection: \"block\"",[107],{"type":25},{"text":109,"type":20}," turns on the AI bot blocking and the Disallow rules that advertise it.",{"type":100,"content":111},[112],{"type":16,"content":113},[114,118],{"text":115,"type":20,"marks":116},"is_robots_txt_managed: true",[117],{"type":25},{"text":119,"type":20}," lets Cloudflare rewrite robots.txt at the edge.",{"type":16,"content":121},[122,124,128],{"text":123,"type":20},"Turning off the first one is not enough, which cost me a round trip. With ",{"text":125,"type":20,"marks":126},"ai_bots_protection",[127],{"type":25},{"text":129,"type":20}," disabled the served file was unchanged. Both settings have to go off, and then the cached copy has to be purged:",{"type":44,"content":131},[132],{"text":133,"type":20},"curl -X POST \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Fzones\u002F$ZONE\u002Fpurge_cache\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  --data '{\"files\":[\"https:\u002F\u002Fexample.com\u002Frobots.txt\"]}'",{"type":16,"content":135},[136,138,142],{"text":137,"type":20},"With ",{"text":139,"type":20,"marks":140},"is_robots_txt_managed",[141],{"type":25},{"text":143,"type":20}," set to false, the file from your repository is served verbatim again. That is the state worth keeping. Robots policy belongs in git, where it can be reviewed, not in a dashboard toggle somebody flipped once.",{"type":33,"attrs":145,"content":146},{"level":35},[147],{"text":148,"type":20},"This is a real choice, not just a bug",{"type":16,"content":150},[151],{"text":152,"type":20},"Blocking AI crawlers is a legitimate position. If your content is the product, you may not want it in a training set, and Cloudflare turning it on by default is defensible.",{"type":16,"content":154},[155],{"text":156,"type":20},"The problem is that it is silently incompatible with the opposite goal. If you want to be the answer when somebody asks ChatGPT or Claude or Perplexity about your niche, you have to be readable. Those two goals are in direct conflict, and the default picks one for you without telling you.",{"type":16,"content":158},[159],{"text":160,"type":20},"That is fine when it is a decision. It is not fine when nobody in the team knows it happened.",{"type":33,"attrs":162,"content":163},{"level":35},[164],{"text":165,"type":20},"The two minute check",{"type":16,"content":167},[168],{"text":169,"type":20},"Fetch your own robots.txt and compare it to the one in your repository:",{"type":44,"content":171},[172],{"text":173,"type":20},"curl -s https:\u002F\u002Fyour-domain.com\u002Frobots.txt",{"type":16,"content":175},[176],{"text":177,"type":20},"If the live file has rules your repository does not, something in front of your origin is writing them. On Cloudflare it is almost certainly these two settings. Check it on every site where being found by AI search matters, because nothing in the codebase will ever reveal it.",{"type":16,"content":179},[180],{"text":181,"type":20},"On this site the two files match again. That took a settings change and a cache purge, and no amount of reading the repository would have told me it was needed.","Cloudflare rewrites robots.txt at the edge and blocks AI crawlers by default. None of it lives in your repository, so reading the file in git tells you nothing.","blog_post","2026-08-31 09:00","robots-txt-you-did-not-write","blog\u002Frobots-txt-you-did-not-write",null,0,[],false,215738016545577,"bee8d07e-97a9-4bc3-ac23-2a7bb249e1df","2026-09-03T12:39:48.600Z","default",[]]