search engines: how do they work?

Learning how search engines work can help you market your site. 

The common misconception is that search engines crawl the web when a user enters a search query. This is not true. Search engines use automated programmes (spiders or bots) to constantly crawl the internet and index websites that they find according to keywords that the website uses. These programmes return to a site and index any changes that may have occurred. These can affect where a website places for search results.  

A search engine has four components, such as spiders, index, relevancy algorithms and search results. Search engines send out spiders (or bots) to crawl as many web pages as possible. These are then read, collected and indexed to give you the best results for your search criteria. The search engine responds to a search query by looking at its own index (not the internet) and returns relevant results for the query. 

Search engines typically want to return relevant and organic results, thus they ignore banners and other rich media (like flash) in the search. PageRank is assigned to the indexed pages and the most relevant results (the highest ranking) get placed at the top of the page.  

All search engines rank relevancy in a different way, using different criteria to determine relevancy. A good idea is to research as many of the criteria as you can and attempt to make your website work on all the major search engines. Another option is to optimise for one search engine, usually the most popular search engine, and see that it works on other search engines as well. 

In terms of SEM (Search Engine Marketing), you need to optimise your website in order to make it work with most of a search engine’s criteria for relevancy. The three largest search engines, Google, Bing and Yahoo!, have guidelines and policies in place that you can read in order to make sure that your website performs well.  

The other way to optimise your website is to hire the services of an SEO company, but make sure that they use white-hat SEO techniques so that you don’t get punished by search engines for bad SEO practices.