How to improve your site search

Making sure that your search engine makes it as easy as possible for your customers to find what they are looking for is business-critical. It is also very difficult - good search engines can cost an awful lot of money and require a lot of ongoing effort to keep them up to scratch.

As an example: on Monday 12th December 2005, I wanted to buy a copy of Jamie Oliver's new cook book Jamie's Italy from amazon.co.uk. So, I went to the "Books" section of their website and searched for "olivers italy" and these 9 items appeared on the results page:

1. "The American Tractor" by Patrick W. Ertel
2. "A Garden in Lucca: Finding Paradise in Tuscany" by Paul Gervais
3. "History in Exile: Memory and Identity at the Borders of the Balkans" by Pamela Ballinger
4. "Oliver Tractors" by Jeff Hackett, Mike Schaefer
5. "Wyoming (Moon Handbooks S.)" by Don Pitcher
6. "Wines of Australia (Mitchell Beazley Wine Guides)" by James Halliday
7. "All Music Guide to Jazz: The Definitive Guide to Jazz Music" by Ron Wynn (Editor), et al.
8. "Larousse Gastronomique: The World's Greatest Cookery Encyclopedia" by Prosper Montagne
9. "The Teacher's Calendar: The Day-By-Day Directory to Holidays, Historical Events, Birthdays and Special Days, Weeks and Months" by Holly McGuire (Compiler), et al.

Jamie Oliver's book didn't appear anywhere on the results page, even though it had been Amazon's 3rd best-selling book in the previous 24 hours.

The problem was that I had typed "olivers italy", instead of "oliver's italy" (which would have returned Jamie Oliver's at the top of the search results list). That single missing apostrophe was all that it took for Amazon's expensive search engine to splutter, fall over and fail.

So - if Amazon can't do it, it must be impossible, right?

Wrong - here are some things the boys & girls at Amazon could - and should - have thought about.

Two types of problems

There are two basic types of problems that a user can experience when they are searching for something:

- User-error - the correct search term is entered incorrectly (i.e. the user intends to enter a search term that would cause the search engine to return results that are relevant to their needs, but they enter it incorrectly).
- Search engine error - the wrong search term is entered (i.e. the user enters a search term that the search engine does not relate to their needs).

User error

People generally enter the correct search term incorrectly because they either:

- Don't know how to spell it.
- Have made a typing error

It's important to realise that there are millions of potential customers who can't spell very well. For example, a 2003 survey of the literacy (i.e. reading and writing) estimated that there were 16% of English adults (aged 16 to 65-year-olds) had literacy levels no higher than those expected of an 11 year-old (source: The Skills for Life Survey).

Also, let's not forget that according to the British Dyslexia Association around 4% of the population are severely dyslexic and a further 6% have mild to moderate dyslexia problems.

This means that your search engine has to account for people making basic knowledge-based spelling mistakes.

Your search engine should also account for people who know how to spell what they are looking for, but make typing errors. The main categories of typing error are:

- Characters close to one another on the keyboard being entered erroneously (either in place of - or in addition to - the correct letter). For example: wrong/wring ; for/dfor.
- Characters being omitted. For example: missing/missng ; oliver's/olivers.
- Characters being entered too many times. For example: impossible/imposssible.
- Characters being entered in the wrong order. For example: disaply/display ; being/ebing.

Your search engine should allow people to make these mistakes and still return useful and relevant results.

Even though we have named these types of issues ‘User error', if your search engine fails to return information that that the user is looking for it is, of course, your fault and not theirs!

Search engine error

When people enter the wrong term into a search engine, it is only wrong because you have not anticipated it. You should aim to cover as many bases and anticipate as many different search terms as possible.

What to do

The next steps for making your search engine perform better are really simple:

- Sit down and make a list of all the spelling errors, typing errors and alternative search terms that you think could possibly be relevant to your site (e.g. actually look at your keyboard and think about what letters are close to one another).
- Ask other people in your organisation to make similar lists.
- Do some research into what search terms people are using on your site (e.g. interviews, questionnaires, check your search engine logs, etc.)
- Apply everything you learn to your search engine.

And that's it. You now have the knowledge you need to begin improving your site's search engine.

Other thoughts

- Improvements in word processing software have made people lazy typists. Software that auto-corrects many spelling and typing errors means that people are no longer forced to review and correct their work to the same extent as in the past. This means that many people are getting out of the habit of precise spelling/typing. So, when they move out of an auto-correcting environment (and onto a website, for example) they are more likely to make - and less likely to notice/correct - mistakes!

- Search results pages should display the search term the user entered in large text (e.g. 28pt). This would help people spot any inadvertent errors.Results pages should also provide the telephone numbers for customer enquiries/assistance.