I recently looked at a client’s robots.txt file for a WordPress site. He had all sorts of things blocked (using the Disallow directives) including:
/*/feed/$
/*/comments/$
/*/trackback/$
As I have written before, robots.txt disallows are generally not as useful as people think they are. True, they prevent Google and other search engine spiders from crawling those URLs, but that doesn’t necessarily mean those pages don’t end up in the index (which is usually the reason for the Disallow directive.) These days we have much better tools at our disposal with the meta robots noindex tag and the canonical.
Let’s say you are a foodie blogger and just posted a recipe on tomato soup. So you wrack your brains – gee what are all the keywords that describe this post? And you come up with “tomato soup”, “healthy eating”, “warm foods”, “easy recipes”. The way WordPress works is (as long as your theme shows tags), those tags will appear as clickable links that will take you to an archive page that show a list of posts that have the same tag. The problem is that unless you have written several posts on “tomato soups” – the tag archive page will just show your one post. Not only is that not useful to your site visitor, but could be diluting your content in the search engine index.
In my quest to make my category pages index worthy, last week I added 
