Monday, November 12, 2012

Using Google Site Search for Content Sites: Things to Keep in Mind

I've used Google Site Search for a couple of unstructured content sites (opposed to applications or structured content such as catalogs) and pretty happy with it, but there a certainly some things you should be aware of before committing to this service:

  • The service's crawler works the same on This means, it will only index publicly available pages that it can access. So you cannot use it on password-protected staging sites.
  • If you consider removing the password from the staging site you won't want to do that either, as your site must be registered with Google Webmaster tools, so your staging site will eventually end up in the index. But there's also another issue.
  • Indexes to URL's contain the domain of your site, so you can't index your staging site, then hope to use that index in your live site.

If you're re-launching an existing site -- your only option is to enable search after you have made that site publicly available via the production domain. If it's a new site, and you have the option, make the site publicly available a couple of days before hand and index the site. Google Site Search provides on-demand indexing, updating the index within 24 hours.

When indexing the site, you have three options for supplying URL's:

  • Individual URL's only
  • URL's linked from a specific page or Sitemap
  • URL's in a Sitemap only

Again, you must have your site registered with Google Webmaster Tools for the indexing to work.

For more information on indexing checkout Custom Search Help documentation.

Please note this blog is no longer maintained. Please visit CivilCode Inc - Custom Software Development.