Systematic Internet searching
 

1. CHOOSE SEARCH TERMS
Make a careful selection of keywords and their synonyms, phrase-in-quotes, sets of terms and phrases, or of likely subject descriptions to submit to the search tools. The following sites provide some background:

Boolean searching on the Internet
Subject vs keyword searching
Creating Your Search Strategy

2. META-SEARCH
A meta-search will give a quick overview and determine which search engines are returning the most hits. It may result in locating exactly what you want, especially if you are searching a unique term or phrase. A meta-search engine does not own a database of  web pages. They use the databases and searching programs of the search tools they query.

Ixquick  translates your search into the appropriate syntax for each of the individual search engines. It supports all forms of Boolean expressions, phrase searching, wildcards, parenthesis and other modifiers such as the proximity operator NEAR. In addition it allows you to search for specific terms in a URL.  Ixquick awards the site one star for each search engine that placed it in its top ten for your search, clearly indicating the quality of the result. It lists each of these search engines and the website's place on each engine's top ten list.
Dogpile searches web sites, Usenet, FTP sites, images, audio/MP3 and streaming media. Choose from the available 14 search engines and set their order so that the more likely engines return first. Boolean searches are supported and response time is very good. Dogpile does limited parsing, e.g. NEAR may become AND, and it supports phrase searches where the individual engines do so. It does no overall sorting other than as received from individual engines.
Metacrawler has three optional interfaces: Regular, Power, which allows you to access all Metacrawler search preferences from the one page, or Low Bandwidth  for those with an old browser or a slow connection. Other options include: Setting the default Boolean operators to be used (OR, AND, or as a phrase); Parse ALL = AND, ANY = OR; Limit to Internet pages from North America, Europe, Asia, Australia, South America, Africa, Antarctica; or limit to U.S. educational, commercial or government sites. The  search channels listed on the main page utilize engines from smaller, subject-specific web sites.
Vivisimo  takes results from established search engines and directories such as Alta Vista and Yahoo and arranges these results in categories. This simplifies the job of selecting those hits appropriate to your particular needs.
Meta-search engines do NOT eliminate the need to learn how to search skillfully using at least a few general web searching tools. The better you know how each works, the better you can judge the reliability of search results.

For further information check the following sites:
What are meta-search engines? When to use, and not to use them,
Meta Search Engine Roundup
The Meta Search Engines: A Web Searcher's Best Friends.

3. REFINE THE SEARCH RESULTS
Choose a particular search engine which has the capability of rating search sets according to factors such as the frequency of occurrence of the keywords, recorded traffic at "hit" sites, or the number of links leading into those sites.

3.1     General search engines
In ordinary search engines or search tools you submit keywords to a single database of web pages owned by the search tool, and you get back a different display of documents from each search engine's unique database of web pages. Where a normal search returns too many hits it is necessary to choose search tools which allow you to refine your topic in order to focus in on the aspects you want.
Alta Vista is a popular search engine which offers the capability for phrase searching and case sensitivity as well as truncation. A particularly useful feature for limiting the number of items found and for assuring relevance is the option of limiting a search to document fields such as the URL, the title, or the domain name.  Use the nearest Alta Vista mirror site for faster searches.
Google is currently acknowledged as the most reliable search engine and provides the most comprehensive results. Google ranks found pages in terms of the number and type of other sites which link to that page.
Hotbot provides extensive advanced search options through the use of query operators and forms. It has more parsing and filtering options than other search engines. It allows you to "Revise" your search, select "More Search Options" and filter previous results by specifying additional terms. Limiting by date is also an option as is the exclusion of specified terms. There is facility to restrict hits by date, by continent and according to the inclusion of different file types such as images and audio.
Ask Jeeves  is one of the few search engines which allows for a natural language search.
AllTheWeb Advanced Search  allows greater flexibility in searching by date (one month, six months, nine months, or a year), it can filter searches by IP address, restrict your results to documents of a certain size (larger or smaller than a user-specified number of bytes, kilobytes, or megabytes) and use fuzzier word queries (words must, must not, or should appear)

Check the following sites for additional information:
How to Choose a Search Engine or Directory This site charts the search capabilities of various search engines.
Search Engine Features Chart gives a detailed listing of features for the most popular engines.
Search Engine Strategies provides a chart to help you utilize the popular search engines.
Spiders Apprentice a helpful guide to web search engines.
Windweaver's Search Helper provides profiles and reviews of the major search engines as well as guides to good searching practice

3.2     Specialised search engines & websites.
In many cases there are more specialised resources which can lead us to an answer more efficiently and effectively than the general search engines.Some examples are:
Amnesi: the Domain Name Search Engine - useful if you have forgotten part of a URL or if you are merely making an educated guess at the address of a site you haven't already visited. It is fuzzy logic enabled to cope with mis-spellings and/or best guess.
MediaBuilder image libraries & online tools - thousands of original web designs, interfaces and photoclips free for use on personal web pages and email.
MESA - Meta-Email-Search-Agent - searches in parallel (nearly) all available sources of email addresses on the Internet.
Northern Light Current News - updates headlines, weather, and sports continuously;  can search two weeks archive.
searchUK - only lists UK and UK-orientated web sites, and provides some powerful features for structuring and filtering your search queries. It is useful if  you are searching for material from British sites difficult to separate from the preponderance of U.S. sites indexed by other search engines.
Speech & Transcript Center - an extensive set of media transcripts- news and public affairs, official and public speeches.
Supersearch for software
SYMBOLS.com - encyclopedia of Western signs and ideograms "more than 2,500 Western signs, arranged into 54 groups according to their graphic characteristics.  In 1,600 articles their histories, uses, and meanings are thoroughly discussed. The signs range from ideograms carved in mammoth teeth by Cro-Magnon men, to hobo signs and subway graffiti". Searching is by word or by graphic charactersitics such as symmetry/asymmetry, curvature, open or closed structure. crossing of lines, etc.

Further information on this approach can be found at sites such as the following:
Choose the best search engine for your information need
Beaucoup! a classified listing of over 2,500 free search sites.
Langenberg - 300 Search Forms in 30 Categories
Country search engines and regional search engines: Links to hundreds of regional and local search engines and country specific Web directories located around the world.

4. CHECK SUBJECT DIRECTORIES
Keyword searching ensures maximum recall but often finds far too many hits to check easily and some of those found have limited relevance.  The Subject Directories on the other hand are usually produced by human indexer/reviewers and consequently exclude much of the ephemeral, the unreliable and the purely commercial sites.
For topics with a lot of web documents out there, you may want to explore one or more subject directories and guides. They contain selected sites, by subject category and are far from comprehensive but they can help you see how your topic is covered in the web.
Yahoo! the most popular subject  directory has a database of about 500,000 sites and is an excellent introduction to topics widely covered on the Internet.
LookSmart contains an excellent collection of Australian websites separable from its international listings.
Ozsearch is a directory which was initially limited exclusively to Australian web sites but now includes the options of searching sites from Singapore, Malaysia, Indonesia or Thailand.
PINAKES: a subject launchpad - impressive array of directories, search engines, databases, multi-subject gateways,  etc.
WWW Virtual Library  The WWW-VL is the oldest catalogue of the web. It is run by a loose confederation of volunteers who compile pages of key links for particular areas in which they are expert. Widely recognised as being amongst the highest-quality guides to particular sections of the Internet.

For additional information see:
Overview of Subject Directories
Internet Search Engines Database(ISED) directory listing
Subject directories [compared]

5. SEEK THE ADVICE OF PEOPLE EXPERT IN THE FIELD
Some of the tools which can be used for this purpose are:
Associations on the Net  The IPL's Associations on the Net (AON) is a collection of over 2000 Internet sites providing information about a wide variety of professional and trade associations, cultural and art organizations, political parties and advocacy groups, labor unions, academic societies and research institutions. Abstracts summarising information about the association and its site are provided.
Australia's Cultural Network
EmbassyWorld Directory and search engine of the world's embassies and consulates
NGO global network  Non-government organisations.
Pitsco's ask an expert
United Nations Homepage
Useful Australian organisations
The World Wide Web Virtual Library: Museums

6. SHARE YOUR PROBLEM WITH OTHERS
Find a mailing list/discussion list or an Internet newsgroup where you can discuss your information needs with your fellow enthusiasts, some of whom may already have worked through the same problem. Since the lists are international, there is often a large pool of participants and there are subscribers online, in one country or another, at all times. Responses are sometimes received within minutes of entering the message on the list.
Google Groups is a specialised database for searching Usenet newsgroups which tend to be for lay audiences rather than for academics.However some academic disciplines prefer them over Listserv type email groups as a scholarly forum. Other mailing list providers and finding aids can be found at the sites such as:
Ozlists: A list of Australian electronic mailing lists (academicially oriented lists mainly)
Yahoo!Groups
People and chat on ninemsn: Communities
Suite101: real people helping real people
msn Communities
Delphi forums
Tile.net  - discussion groups (mailing lists) using the Listserv software, Listproc or Majordomo software

There are a number of sites designed to assist those who are new to this resource including:
How  to use mailing lists
How to use Newsgroups
Listserv and Newsgroup tutorial
Sources for Information About Mailing Lists from PAML (Publically Available Mailing Lists)