Google’s new robots.txt video says the standard covers crawling, not how content gets used, and recommends percent-encoding non-English robots.txt paths.
The post Google Explains The Limit Of Robots.txt For AI Content Use appeared first on Search Engine Journal.