Sadly, every so often, you hear of content being indexed and exposed on Google and other search engines. Often the issue is not necessarily with the search engine but with the site that hosts and holds that information. Those sites often do not set the proper blocking mechanisms in place to communicate to Google and other search engines that the content should not be indexed and shown within the search results.
That is what happened recently with Anthropic’s Claude Chats showing up in Google, Bing and other search engines.
More details. Wired’s story named Private Claude Chats Exposed in Google and Bing Search Results explained how private chats from Claude were found on the web on Google, Bing and other search engines. The chats included politics, health discussions and more, all pretty sensitive chats. “Claude allows users to share with other people “snapshots” of chats by creating a public URL to a specific chatbot thread,” Wired explained.
“The reasons some of these URLs were indexed by major search engines comes down to the basic functions of websites, search engines, and the collision of the two when generative AI gets in the mix,” Wired added.
The primary issue is that if you block both using robots.txt directive and use noindex on that page, search engines like Google and Bing won’t be able to see the noindex tag since the crawlers won’t be able to access the page with the directive.
Doing a site command for [site:claude.ai/share] would return hundreds of chats from Claude over the weekend, but now those results have been removed.
Glenn Gabe said on X that he wished these journalists would have spoken to an SEO before covering the story. “It’s filled with bad information. If you block via robots.txt AND noindex the page, Google and Bing *cannot* see the noindex tag since they can’t crawl the page and see the tag in the HTML,” he explained correctly.
This is not new information; Google’s own documentation has a huge notice at the top of the page that says in bold and red highlights:
“Important: For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”
What Google said. Ned Adriance, a Google spokesperson from the Google Search side of the team, understands how this works. And he told Wired, “Neither Google nor any other search engine controls what pages are made public on the web, and these pages were indexed across many search engines.” He added, “We give site owners clear controls to decide whether pages can be crawled or indexed, and we always respect those directives.”
For some reason, Microsoft Bing and Anthropic did not provide any comment to Wired on the issue.
Why we care. This shows the importance of consulting with an SEO who understands how to ensure the right pages are indexed and visible in search and maybe more importantly, the pages you do not want to show up in Google or Bing Search are not discoverable and do not show up in the search results.
Controlling the visibility of your content is SEO 101, and sadly, we see this issue come up over and over again with sensitive information being open and available on the web.