|
Search
Engines
A search engine is basically an online
application which web surfers use to find web sites
and documents using a query of normally two or three
keywords. The best known examples of these are Google,
AltaVista, Fast/AlltheWeb and Teoma, although Google
is by far the most popular. One that the general public
may not know but is none-the-less important is Inktomi,
which provides the search results for several Web Portals,
notably MSN and up until a couple of years of ago also
powered the secondary results of Yahoo.
Search engines are defined by the way they build their databases.
A search engine runs free crawling applications which are programmed to start
at a defined point on on the web, cache the data it finds, and then follow links
to other documents to do the same. These applications are commonly known as robots
or spiders and can be seen in your raw log files under a variety of user agents
or identities. Once the spider has finished its crawl, the data is measured against
the search engines unique and secret algorithm and each document is compared and
scored.*
*This is a generalised definition.
SEs operate in different ways and exact methods are
known only to specific SE employees.
For more information on how to improve
your website for high rankings in search engines take
a look at our section on site
optimization .
So Who Are The Major Players?
OK. First thing to remember is that
many SE's are partnered with and often rebranded in
many favourite Web Portals. For instance the backbone
for the BBCi Search Engine is powered by Google. So
the reach of some engines is huge even though web surfers
may not necessarily directly search from the SE's home
page.
Teoma is
one of new boys in search technology "Teoma was
founded in 2000 in Piscataway, New Jersey by a team
of scientists from Rutgers University. Teoma means "expert"
in Gaelic. Ask Jeeves, Inc. acquired Teoma in September
2001. "
Teoma is an excellent search engine
with the potential to become as popular as Google. It
has a free crawling spider and it's index is updated
fairly regularly. There is also a PFI service if you
want to get your site listed quickly and spidered regularly.
More details of this, see resources opposite.
Teoma now powers the search results
of Ask Jeeves. Teoma's spider can be found in your raw
logs under the user agent of "TeomaAgent".
Google was
launched in 1998 by Stanford University graduate students
Larry Page and Sergey Brin. 4 years on it has become
without doubt the most popular search service on the
Internet and drives more search based traffic than any
other website, approx. 150M searches per day. Some webmasters
have reported over 90% of their traffic as generated
by Google even when their site is equally well listed
elsewhere.
Google powers search for many Web
Portals, notably AOL, NetScape, Yahoo and LookSmart
UK. Its spider goes under the user agent identity of
"Googlebot".
Fast is
a Norwegian based search company that has quickly built
a name for itself. It first came to being in 1999 and
claims one of the biggest databases of web documents
and as the name suggests is incredibly fast. The search
engine is built on very strong and reliable technology
and search results are generally excellent. Fast's figurehead
site is AlltheWeb
which is not well known by the general public. Although
erratic in recent times Fast spiders do crawl the web,
but if you want guaranteed inclusion, Fast's PFI service
will list you in 2 days and your pages spidered every
48 hrs.
Fast powers the search results of
much of the Lycos network.
Inktomi was one of the pioneers
of search technology. Although not a search engine per se, it powers the results
for a number portals, notably MSN and Overture. Inktomi's database is built on
a combination of paid and free listings. Paid listings are via partner sites such
as Position Technologies and free crawling
using its spider "Slurp".
Inktomi looked like they were pulling out of the search service
industry a while ago when they lost their contract with Yahoo which quickly led
to further losses with other partners. However their partnership with MSN still
appears to be strong and with a 2 week update cycle and rumors they are chasing
new contracts, Inktomi are looking stronger than they have done in the past few
years...
|
