So, I have a web-site with some automatically-generated pages. It includes very mildly personal information; for the sake of argument, let’s say it is their shoe-size.<\/p>\n
This information is intended only for a select group of shoe-enthusiasts, doing research. I don’t want this web-site to appear in the search engines. People randomly searching for someone shouldn’t find their shoe-size. <\/p>\n
The first option would be to password-protect it (or otherwise authenticate each user). However, that is hugely onerous on the legitimate users of the site, for something as mild as shoe-size.<\/p>\n
So, I went for the honour system provided by robots.txt. “Please, please,” I asked the search engines, “Don’t go looking in my shoe-size directory.” While it is not guaranteed, the convention robots.txt is widely followed by bots.<\/p>\n
Google honoured the letter of my request, but not the spirit. (I only pick Google out because I use it, not because it is necessarily the only offender.) It doesn’t search the page, but when someone links to it, Google will still return it. (e.g. How do you get around this? Google explain<\/a> that there is another convention – the noindex meta tag<\/a>.<\/p>\n I need to put in every page a note to the robots to say “Hey, don’t include this very page in your index at all.” But, the only way that they will see that note is if I let the search engines read the file – I need to remove the robots.txt restriction.<\/p>\n Now, I am in a dilemma for two reasons.<\/p>\n The first is that I have to let Google crawl all around my database, which may have hundreds of thousands of records and reports of shoe-sizes. That is going to cost me in bandwidth and CPU, just to say “This page shouldn’t appear in your list”. <\/p>\n The second is that there are sure to be robots out there that support the robots.txt convention but not the newer noindex meta tag convention. I don’t know who they are. I can’t exclude them in the robots.txt, while allowing the bots through that comply with the new convention.<\/p>\n We need a updated standard for robots.txt that says “Not only shouldn’t you look in this directory, you shouldn’t even admit that such URLs exist in your database.”<\/p>\n","protected":false},"excerpt":{"rendered":" I am in a dilemma about how to tell Google to not index my pages.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"","footnotes":""},"categories":[25,34],"tags":[114,265,266,264,95],"class_list":["post-531","post","type-post","status-publish","format-standard","hentry","category-insufficiently-advanced-technology","category-software-development","tag-google","tag-html","tag-robotstxt","tag-search-engines","tag-web"],"_links":{"self":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/531","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/comments?post=531"}],"version-history":[{"count":0,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/531\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/media?parent=531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/categories?post=531"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/tags?post=531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<a href=wherever.html>Kevin Rudd's Shoe-Size<\/a><\/code> will be found when searching for Kevin Rudd’s shoe-size, even though the URL itself is never visited by Google’s bots.)<\/p>\n