Using search engines

The good news is, all the major search engines do a good job of cataloguing a fair slice of the Web as well as other sections of the Internet. Most of them also work in basically similar ways, at least at the simple level, so you can transport many of your basic searching skills from one engine to another.

Search engines distinguish themselves in a variety of ways: some catalogue more of the Internet than others; some stay more up to date; some are easier to use; some add very advanced searching options. There's an incredible amount of jostling among the engines to be number one, so you'll find frequent upgrades to their interfaces and underlying technology. The engine that was hot three months ago may already be slipping out of favour today.

Search engines are divided into two main types: those that use a keyword search approach and those that categorise the Web into a series of hierarchical topics.

To use the keyword engines, you type a single word or phrase and the search engine brings you back a summary of pages or resources which contain a match. You then browse through the results list and, when you find a likely looking document, you click its link to go directly to the Web page.

Probably the easiest to use and most effective of all keyword search engines is Google.

To use the category searches, you use a drill down approach, starting with general topics (such as The Arts, Business, Computers) and digging down to specific categories (such as Chinese Poets, Debit Cards or Electronic Mail Software) refining your search as you go.

Most of the keyword engines attempt to index almost every word on every page of the World Wide Web, a Herculean labour which usually means they're somewhat out of date and, in all cases, that their job is incomplete. This approach makes them very comprehensive, but often means you'll find lots of 'broken links' (Web pages that have disappeared or moved since they were catalogued).

As some search engines use an 'index everything on sight' method while others categorise only those pages submitted, you'll find huge discrepancies in the scope of information you can locate with any particular search engine. Some engines also index information in newsgroups, FTP sites, e-mail directories and from other sources, giving them a much broader scope than mere Web-based searches.

If you're looking for obscure or highly specific details, use one of the keyword engines, such as Fast or Google. To use these search engines effectively and not be totally overwhelmed by the amount of information they can toss at you, you'll need to take a little time getting to know the specific syntax they use. Each search engine uses its own search language, and the syntax varies slightly from one to another. Taking some time to learn the specific syntax for a search engine will increase the value of the results you retrieve many times over.

If you're wanting information about more generalised topics, such as 'What sort of handheld computers are available?', then you'll find the category-based search engines are ideal. For example, to answer this question using Yahoo, you'd start by looking in the computers category, then hardware, then portable computers, and finally handheld computers.

The most effective way to use search engines is to choose two or three (include at least one keyword engine and one category engine) and begin to use them regularly. That way, you can become accustomed to the particular features and language used by the search engines without drowning in the sea of possibilities.

Meta-search engines, such as Dogpile and IXQuick offer another option. These search engines take your search query and submit it to a selection of other search engines, including both keyword and category engines. Meta-searches usually restrict the number of results either by limiting the amount of time they spend on the search, or by limiting the number of documents they'll return in the results list.

Localising your searches

If you want to focus on sites in your neighbourhood, try a local search engine or a local branch of one of the major search engines. Yahoo, for example, has fourteen national Yahoos in addition to the main US-based site. These local search engines let you choose between narrowing your geographical focus or doing global searches.

You can also use one of the big global search engines and restrict your searches to local sites by doing field searches. Field searches look for the terms you specify in particular parts of a Web document. For instance, to restrict searches in AltaVista to Australian sites, add AND host:au to your query.

We've focussed on search engines that primarily search the Web. You'll find there are engines that also search FTP sites, newsgroups, mailing lists and specific databases. There are search engines for e-mail addresses, individuals, and organisations. There are engines which focus on different geographical regions or particular languages.

If you're after a more specific search engine, check out the list of specialty search engines at Search Engine Watch. You can also find an impressive list of engines simply by typing search engines in the Yahoo! search box.

George Boole was a British mathematician who suggested logical thought could be expressed as algebra. One of his legacies is the logical terminology recognised by most search engines. You don't have to use Boolean language when using search engines (some don't support it at all) but if you know how to use it you can gain far greater control over your searches.

The main Boolean operators are OR, AND, NOT, and NEAR. Some engines also add ADJ, FAR and BEFORE. It's easiest to understand what these terms do by looking at some examples.

OR searches for documents that match either search term.

Example: sheep OR dip will find documents that contain references to either sheep or dip or both. This search could turn up pages referring to sheep, to sheep dipping, to food dips (such as French onion dip), to dipsticks, and so on.

AND searches for documents that match both terms.

Example: sheep AND dip will find only those documents that refer to both sheep and dip. AND gives you much more focussed results than OR.

NOT searches for documents that do not contain the specified term.

Example: sheep OR dip NOT guacamole will find those documents that contain a reference to sheep or to dip but which do not mention guacamole.

NEAR searches for documents that contain the specified words within the vicinity of one another. What constitutes the 'vicinity' varies from search engine to engine. It's usually from 10 to 25 words. With some engines you can specify proximity with the syntax NEAR/n, where n is the maximum number of words which can separate the two search terms.

Example: sheep NEAR/10 dip searches for documents in which sheep appears within 10 words of dip.

ADJ searches for documents that contain the specified words right next to each other, in any order.

Example: sheep ADJ dip will find documents containing 'sheep dip' and 'dip sheep'.

FAR searches for documents in which the search terms are 25 words or more apart from one another.

Example: sheep FAR dip searches for documents containing both sheep and dip but separated by at least 25 words. Thus, this search will exclude any documents that contain the term 'sheep dip'.

BEFORE works just like AND, except that the terms must follow the specified order.

Example: sheep BEFORE dip looks for documents containing both terms, with sheep preceding dip.

As some search engines require that you type the Boolean operators in upper case it pays to always abide by this convention.

When you use more than one operator in a search query, you can ensure the search engine analyses your query in the order you want by surrounding terms with parentheses. For example:

AltaVista

1. The Help offered is different depending on whether you're using the Simple Search (default) form or the Advanced Search form.
2. Use quotation marks to search for specific phrases, such as "sheep dip". (The same technique applies to many, but not all, search engines.)
3. You must use AND NOT instead of NOT to exclude items in Boolean searches. This is contrary to the information in AltaVista's online help. The same applies to Excite and HotBot.
4. Use the + sign in the Simple Search box to ensure a term appears in the document; use - directly before a term to exclude it from a search. For example, peach apple pear will turn up documents containing any one, two or three of these fruits. On the other hand, peach +apple pear will find documents containing apple that may also contain peach or pear. And peach -apple pear will find any documents that contain the words peach or pear, except those documents that also contain the word apple. Make sure you don't leave a space between the + or - sign in the search phrase.

Google

1. Google tries to make searches as simple as possible by automatically searching for documents which contain all terms you type. For example, if you type sunken treasure titanic, Google will put documents containing sunken and treasure and titanic at the top of its results list.
2. Google tends to toss out small common words such as when, to, in, the and so on. So don't waste your time using them.
3. If you want to match an exact phrase, place it in quotation marks: "freshwater pools".
4. Check out the tabs near the top of Google's search page. You can search for images, discussion groups, news headlines and more with Google.
5. Click the Preferences tab to change how Google presents its results.

HotBot

1. Use HotBot's Advanced Search mode. It's all form-based, not particularly difficult, and very powerful.
2. Can't find what you want? Click the Second Opinion link at the bottom of the results list to see whether Lycos can turn up better results.
3. Narrow down search results by clicking the Search Within These Results option at the top of the results list, then click the Search button once more.

Yahoo!

1. Use your national Yahoo!, if there's one available. They give you all of Yahoo! with a local bias. You'll find a list of them at the bottom of Yahoo!'s home page.
2. Yahoo! is chiefly a well-categorised directory, rather than a search engine. Using a drill-down approach through Yahoo!'s categories will usually give you better results than a keyword search.
3. Use + to specify words that must appear in a document; use – to specify words that must not appear in the document. Several other search engines, including Google, share this syntax.