SEO 101 - Part 3

How search engines work

Just because your site has been crawled, doesn't mean your site will be indexed by the search engines. Lets take a quick look at how search engines work. There are 3 stages - stage one is where search engine crawlers visit websites collecting the information on webpages and following links on those pages to other pages to collect the information on those pages and so on and so forth. On some websites the crawler may decide to only crawl a few pages if the site isn't considered too important (and we'll come back to this later, as that 'not so important' site may be yours!).

After collecting all that information, usually hundreds of millions or even billions of pages, the search engine starts indexing those pages. The search engine analyses each and every page to see what the page is about and how it fits in the site and the overall world wide web. The search engine decides which search queries the page may be relevant to, and then ranks that page against all the other pages on the web which are about the same topic. A single web page may be relevant to few or even dozens of different search terms so these all have to be taken into account. Obviously this is quite a simplistic view of how this works but I just want to give you some idea of the enormity of this task.

When a searcher types a search query into the search page, the search engine searches through it's index for what it thinks is the most relevant pages which will match the searchers request. This typically (and quite amazingly) only takes a fraction of a second, and the searcher is presented with a list of webpages that the search engine believes matches their request. For example, I just did a search for "home loans" and there were 34,000,000 results returned. Consider for a moment that you would like your webpage to rank in the top ten for the term "home loans", your page would have to better than 34,000,000 others! Of course, that is not impossible but unless you are a large national or multinational home mortgage company, there is no need for you to rank in the top ten (from a searcher's viewpoint).

Okay, this is where we get to part about why your pages aren't indexed. Even though your page exists, that does not mean it will be automatically included in the search indexes. There are far too many pages on the Internet to include every single page. Search engines need other clues about a page's quality to include it.

Next... What you need to do to get your pages indexed - and start ranking!