Category Archives for Search Engine Optimization

Search Engine Meta Tags – no index

When we first build a website, the thought of actually telling Google and the other search engines to not spider a given web page seems counter-intuitive, why would anyone want Google to not spider their website? (Well except when you are Rue La La).

Here’s one reason. A more sophisticated website might have a login page or registration page. Often these pages shouldn’t be indexed as they don’t add value for ranking for keywords. Compounding the issue, in one case I looked at, the registration page was manifesting as many registration pages, because the site was tacking on a return url in the parameter (so that after the registration the user would be returned to the calling page), creating duplicate content.

If you have many URLs that all point to the same page, that is known as duplicate content (this is different than duplicate content across many websites … and worse) and definitely to be avoided. Each site gets limited link juice and a limited spider crawl budget, you don’t want to waste either on yet another version of a page the spider has seen before.

So to tell the spiders you don’t want a page to be indexed, you put the no index meta tag into the HTML source code (between the open and closing <HEAD> tags) for that page.


Why the follow? So that the link juice from external incoming links and internal links can pass through to the links on the page you are noindexing. Otherwise you are creating a dead end that stops the link juice from passing through. The registration page might not be important, but it might have links to articles that are.

I wanted to explicitly point this out, because if you search on “meta tag no index” you will find lots of examples of “no index, no follow”. Lindsay Wassell makes a compelling case for the right use of this tag in her article and explains why using robots.txt instead is not a viable alternative.

More on the noindex meta tag.

Google’s shaking things up

Have you noticed that Google has introduced some major changes this year? I used to think that if you did the right things: 1) have a well optimized site and 2) build quality links to that site, that was all you needed to care about. While that is still true, the Google landscape is rapidly evolving and we need to pay attention.

First we saw the May Day update (appropriately this update occurred around May 1st, 2010). While Google is constantly tweaking it’s algorithm, periodically there are major algorithmic changes that many noticed. In this case, the major impact was to the long tail search. It used to be that you could rank pretty easily for a long tail search by just slapping a website up optimized for it. Long tail searches are those searches with many words (ie. four, five, six words). With May Day, that’s not as easy to do, as Google now pays more attention to a site’s authority and quality even in the less competitive long tail arena.

About the same time we also had Google Caffeine. This was an infrastructure change for Google that allowed its search spiders to work faster, so “fresher” results could be provided to the searcher. Webmasters noticed that more pages of their sites were being crawled by Google. As Google had warned it would in the previous year, it started paying a lot of attention to page load speed. If you have Google Webmasters Tools set up for your site, check out the page load speed report (under the “Labs” menu) … is your page speed faster than at least 50% of the other sites on the web?

Have you noticed that Google is now providing results as you type into the search box? This is called Google instant. The question then becomes whether this will impact searching behavior. I’ve noticed it impacting my searching in certain cases, especially if I’m not clear on what I’m looking for. If I see that there is a particular subtopic that I hadn’t thought about but is relevant to what I’m looking for, yes I would be likely to chose the search Google is suggesting.

Local business should take note of the more recent change. Google Places (previously named Google Local Business Center) has just had a major update. The most obvious change is no more “7 pack” (those results that look like a list of map locations). The map and the little balloons are still there but the “local” results now look more like the rest of the results. This does mean that the “local” results take up more of the page, signifying Google’s increasing commitment to local business. If you are a bricks and mortar business and haven’t paid attention to Google Places before, you need to know. And those of you that are a local business but not a bricks and mortar, you can still list in Google Places with its new concept of “Service Area”.

So what are the takeaways from all this? Here are mine:

  • Google is committed to local business, if you don’t have a website or even just a web presence, it’s time to fix that
  • Quality and Authority are important, Google continues to wage war on those that would try to game the system with spammy sites. Sites that have unique and fresh content and that demonstrate authority (by having other high quality and relevant sites link to it) are favored by Google.

I think the other thing I would like to point out here, is I think Google’s philosophy has shifted here. It used to be about the sites that were the most popular, now Google is recognizing that many searches have commercial intent and catering to those searches.

Additionally it is providing different types of results mixed in on the results pages. There is the listings for local businesses as we discussed above. But have you noticed as well “Google Shopping Results” and “News for …”?

Search results for Digital Cameras

In the results above for “digital cameras” note the “Brands”, “Stores” and “Types” links. I’m not sure when these links crept in, but interesting eh? Particularly for those selling into a niche market, the “types” links would be worthy of further investigation.

Customizing Your WordPress URLs

With any new blog I work on, one of the first things I do is change the permalink structure. WordPress permalinks will determine the URL that your post will appear under. Just go to any blog, click on a post title and then look at the URL shown for it. If the blog is using the default permalink structure, you will see a question mark and some numbers. Since keyword rich URLs is one way to optimize your blog for the search engines, why not change your permalink structure to a more search engine friendly one?

In most cases I use the custom structure of /%postname%/ (see below caveats on when you might want to use a different one). You can set this up for yourself under Permalinks in the Settings Menu.

Type in "/%postname%/" for your custom permalink structure

This means (as a default) my post title will also comprise my URL (with dashes in between the words). You do also have the ability to customize your permalink to be different than your title. I often do this because I want my blog posts to have catchy titles, which doesn’t always mean that they are optimized for keywords.

Now here are the two caveats you need to keep in mind:

  1. If your blog has been around for a while, proceed with caution with changing your permalink structure. Your existing blog posts may have already been indexed by Google with the old URLs, and other sites may have linked to them. In this case both Google and those sites would get broken links. There is a way around this, but the one avenue I’ve looked at (mod rewrite code in the web server) is non trivial and not available to many. More research needed. Update: there are plugins that will set up the 301 redirects for you
  2. If you think you are going to have a lot of pages & posts (more than 50), read through this forum discussion, you may want to consider /%year%/%postname%/ instead. My blog has a lot of posts, so I may need to consider a change, however point 1 will apply so I will need to proceed with caution.

Letting the Search Engines Know About Your Site

Forget those “search engine submission” services (what do they do anyway?) Here’s how to notify Yahoo and Google about your site and, even better, give them a roadmap to all your pages. I recently had to do this for three websites, so I captured the process of submitting a sitemap for both Google using Google Webmasters Tools and Yahoo using Site Explorer.

I’m assuming you have already built a XML sitemap of your site. If you haven’t and you have a wordpress site, see my 4 essential plugins post, if you have some other type of site, check into gsitecrawler..

Before you can submit your sitemap to Yahoo and Google, you have to “verify” that you own the site. You’ll need an account (ie. a gmail account for Google Webmaster Tools) and a way to modify / create files on your site. Note that there is multiple ways to verify your site (4 for google / 2 for yahoo), so I am just going to cover the uploading HTML file for verification method.

Step 1: Add Your Site Create an Account or Login into each with your Google/Yahoo accounts:

For both the first part of adding a site is straightforward and intuitive. For Yahoo it is under “My Sites”.

Adding Website in Google Webmaster Tools

Adding Website in Google Webmaster Tools

Step 2: Download / Upload the verification file

Step 2 is more complex. When you add your site you will be prompted to go through a validation process. There will be several steps to this:


Google Verification Steps

Google Verification

    Just follow the steps as outlined:

  • First download the file – it will be a long filename that starts with “google”
  • Then upload to your web server’s root directory using a FTP program. Note that you will need to know your hosting account details (which is likely not the same as your wordpress login details)
  • Check that the file is there by clicking on the link, you should see something like “google verification” in your browser
  • Now click on the verify button

For Yahoo the process is very similar:

  • Download
  • Upload
  • Confirm link in browser
  • Click Authenticate
Yahoo SiteExplorer Authentication

Yahoo SiteExplorer Authentication

Note that with yahoo the filename starts with “y_key”. With yahoo I had failures with the authenticate step. It might have been because my blog has a redirect in .htaccess to redirect traffic to my blog directory (although I added the files to both the root and the /blog directory) . It might have been because I first added the site as rather than In any case, persistence was the key, I deleted the site, readded the site as and tried the authentication twice, the second time it seemed to go through. Note that with yahoo, it sometimes just sends you back to the initial rendering of the authentication page, without telling what happened.

Step 4: Add your sitemap

To add your sitemap you will need to know it’s filename. Google XML Sitemaps plugin usually defaults to calling your sitemap: sitemap.xml.

In Webmaster tools, the sitemap options are on the lower right hand side for your site. SiteExplorer calls sitemaps “feeds”.

Site Explorer Feeds Menu Item to add Sitemap

Click on Site Explorer Feeds Menu Item to add Sitemap

A note about Bing:

Even though Bing and Yahoo are merging due to the Microsoft / Yahoo search engine deal, it might not be a bad idea to submit your sitemap to Bing, fortunately it’s easy:

Substitute your Sitemap URL into the string below (eg. and run it in your browser URL box. URL of your sitemap here

You’ll get a “Thanks for submitting your sitemap.” message in your browser window.

SEO Chat Keyword Difficulty Tool

If you want to get a sense of how difficult it would be to rank for a particular keyword, the tool below from SEO Chat will help you out:

Keyword Difficulty Check Tool © SEO Chat™

Enter a keyword or keyword phrase

Enter Captcha To Continue
To prevent spamming, please enter in the numbers and letters in the box below

Report Problem with Tool.

Google mayday update

Google is constantly tweaking and refining its algorithm, but a couple of times a year a significant enough change is made that webmasters really notice. This most recent change, called the “mayday” change due to it’s timing (around May 1st) impacted long tail searches the most. Some sites noticed, some did not. Matt Cutts in the below video explains more:

301 redirect for wordpress domains

While has given you a way to blog under your own domain using custom mapped domains …. it hasn’t been a perfect solution. If you set it up right when you started blogging, there is no issue, your content gets out on the net under your own domain (eg. rather than But for those of us that have a subdomain well established under it was less than ideal. Why? Because the redirect from to was a 302 redirect rather than a 301 redirect.

302 redirects, which means “temporarily moved” are treated by the search engines in an unpredictable manner, your original URL might get indexed or your new one .. who knows. In contrast, a 301 redirect, “moved permanently”, created a much better chance of your new URL getting the index and the traffic.

wordpress has fixed this now though, now the redirect from your subdomain to your custom domain is now a search engine friendly 301 redirect. Now, there is no excuse for me not to create a domain for my real estate blog.

Google Local Business Center for Home Businesses

Adding a listing to Google Local Business Center helps your local bricks and mortar business get found by web searchers. Google Local Business Center is integrated with Google local search – that list of results that show up as a map with locations. If you type in a search term plus a geographic term such as “Thai food Fremont” you will often get these “map” results.

Google Map Results

Google Map Results

The problem has been that you needed a business address for this to really work for you. Many home based businesses don’t want to publish their home addresses and their service area may be different than where they live. Some home based businesses have gotten PO Boxes just to get a better listing in local search. But a pin on a map that points to a PO Box isn’t necessarily that useful.

Well a fix from google now available. In an indepth interview with Eric Enge, Carter Maslan shares the concept of service areas that google is introducing. If you are a dog walker that serves an area differently than where you live, this update is for you. You can now specify what area you want to appear in for a local search. Check out the service area help page for more info.

How do search engines use meta tags?

Part 2 of What are Meta Tags?

We now know what are meta tags (previous post) and that they are really only two that we care about (well most of us, there are some additional ones that should be in a webmasters tool kit). So how do search engines use meta tags?

Keywords meta tag:

It used to be that you could put the keywords you wanted to rank for into the keyword meta tag, and the more the better right? Then you could sit back and consider yourself done with your SEO efforts. This has not been true for a while. Google completely ignores this tag and has for years, and while yahoo and bing probably look at the tag, it is not clear that they give it much weight in determining how to rank your site. Regardless, do not put 30+ keywords into this tag, this is known as keyword stuffing and will only harm your page. Keep it to 10 at most. If you do use adsense or other context driven ads on your site, there is evidence that they use the tag to determine what ads to show.

Description meta tag:

Technically this meta tag doesn’t help your web page rank any better, but it is still important. Why? Because if this tag is set, google will often show it as the snippet underneath the title in the SERPs (search engine results pages) rather than grabbing some random chunk of text off your page. I’ve seen some funny snippets over the years, usually from sites that have no text, so you get a snippet that says “you need flash version xxx to view this site”. Is that going to entice you to click? You can think of your description meta tag as the “welcome” mat to your site. Put together a catchy description (no more than 160 characters as that is all that will show) that will invite your reader to click through to find out more.

What are meta tags?

I belong to a mastermind group that is following a Joel Comm video series. We get together every week and discuss the particular session we watched during that week. Some of his videos do assume a certain amount of knowledge especially when they veer off the prepared materials.

The discussion was concerning meta tags and how important (or unimportant) they are for SEO. I was glad I watched because I picked up one interesting tidbit about meta tags that I hadn’t known before .. but I’ll get to that in another post.

So what are meta tags?

First off, it’s useful to understand that what you see on a web page isn’t the whole story, there are certain HTML code elements that are not visible on the rendered page but still read by search engines. Meta tags fall into this category. Meta tags are metadata, essentially data about data (if that is confusing, I’m afraid the wikipedia entry won’t be much help). However in this specific example, the concept is a little easier to grasp, since on an HTML web page, meta tags help describe what the web page is about.

What can you say about a web page? Well a number of things, but you need to only pay attention to two tags “description” and “keywords”.

  • Keywords is just that, a list of keywords that you think describe your page the best
  • Description is a few sentences that describes what your page is about

Many website building tools and CMS (content management systems) will have some way for you to fill these tags in, without having to learn how to code them into HTML. If they don’t, consider moving on to one that does.

Next – how search engines use meta tags