Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern

Google’s new robots.txt video says the standard covers crawling, not how content gets used, and recommends percent-encoding non-English robots.txt paths.

The post Google’s Illyes: Robots.txt Covers Crawling, Not AI Training appeared first on Search Engine Journal.

https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/