Google On-Demand indexing for Custom Search

Posted by Matt Thommes on 11/18/2008

If you think Google has already infiltrated every corner of the web, think again. There's still plenty of real estate available.

Your own website may benefit from Google's Custom Search results. In a nutshell, you can embed Google search results directly within your page layout and design. The results appear as though they are physically part of your site, rather than being directed to a separate Google search results page.

Pain in the Tech currently uses this approach for search. The results you see are equivalent to going to Google.com and typing:

site:paininthetech.com my search query

... except they reside within the page layout.

The benefit to using Google Custom Search is that you don't need to bother building a search results page! The results are provided for you, fresh from Google's index, in any spot on your site that you wish.

However, there is one thing to be aware of. When I said "fresh from Google's index," I was lying somewhat. Brand new posts on your site (posted same day) usually won't show up in Google's index for a short while - perhaps 24 hours or longer. This is not a major problem, but surely you'll want your local search results to reflect everything in your database, right? It sounds like using Google Custom Search may not be so great after all.

Enter On-Demand indexing - the solution to the problem. Instead of having to wait 24 hours for new posts to show up in your embedded Custom Search results, you can now request that new posts get indexed immediately.

All you need to do is provide Google with a Sitemap, which is typically an RSS file with the most recent content. From then on, your Custom Search queries will take into account the Sitemap with the most recent content, further strengthening your search results.

About the author(s)

Matt is an independent publishing enthusiast, mobile blogger, content creator, informative writer, web developer from a suburb of Chicago. Never one to conform, Matt intends to promote the effect the web has on our lives, in an effort to intensify, instruct, and clarify all that is happening around us.

Comments

Note: Comments may be viewed by authors, but if you have a more specific question you'd like to ask them, please email matt.thommes@paininthetech.com.

# Vishal Shah at 11/18/2008 11:19 am cst

Sitemap's can use RSS (or ATOM for that matter) Matt, but that's generally not recommended as it will not expose pages other than the articles referenced. Its more of a shortcut to enable sitemaps for your site.

If possible, the sitemap should be generated automatically using say your site's feed as one source, along with other webpages on your site mashed in - for example your about pages. As an alternative, you can have a top level sitemap that links to other sitemaps (link) - and one of the linked files could be the syndicated feed. This can be a very effective solution, since there is little manual work needed (as the syndicated feed is generally auto created from a cache or the data store)..

Also note that with RSS, you will not be able to specify Google or other search engines, properties such as priority and changefreq, which serve as important hints to the crawler system.

Vishal Shah

Comment preview: