Bot Preference Sync sets AI crawler policy per category, not per crawler. Allow GPTBot and block Bytespider, and no setting describes what you do.

The post Cloudflare Will Write Your Robots.txt, And It Has A Point appeared first on Search Engine Journal.

https://www.searchenginejournal.com/cloudflare-will-write-your-robots-txt-and-it-has-a-point/589262/