Until a few years ago, webmasters could use the “noindex” instruction to request that Google does not index a page. Usually this instruction was given through a metatag called “robots” in the <head> part of a page.
Google would not index a page where it found the tag “noindex”. This meant Google was not adding to its index any link between the words in the page and the page itself (as happens in a book index). Without indexing, the user was not able to find the page with a query to the search engine.
However Google has changed the “noindex” instruction in recent times. Today Google is even indexing pages with the “noindex” tag, but they are not shown in the search results.
In other words, the “noindex” tag affects the visibility of the page in the search results, but is not affecting what Google actually indexes. A consequence of this is that Google can use resources in noindex as it wishes.
This is something to consider, especially when we don’t want to allow Google to understand what a page is about. The “noindex” instruction is no longer sufficient. The appropriate alternative is to ask the spiders not to download the page through a “disallow” instruction in the robots.txt file.
This post is the English translation of a chapter of the italian ebook “Mitologia SEO” written by Enrico Altavilla.