Search Engine Robots - Search Engine Spiders Explained

Idiots Guide to Search Engine Spiders

Don't be afraid of the spiders! The main spidering search engines, Google, Yahoo and MSN, use programs called search engine robots or search engine spiders to find web pages to build their index. (You may also hear them called, index robots, search spiders, indexing robots and various combinations of these terms). The search engine spiders are employed 24 hours a day, 7 days a week, to ensure that their index is kept up to date.

The search engine spiders are not limited to a pre-defined list of web pages to look at and to index, they simply follow links on each page, that they index, and then they follow the links on those pages, etc. The process of following links is called spidering .

Search Engine Spiders Index as they go

The search engine robots come onto your index page from a link from another site. It stores all the information on your index page and then follows the links on that page and then indexes those, then follows the links on those pages, and so on. That's how the search engines builds its index of millions of web pages.

Spidering Issues

Search engine spiders sometimes have problems finding pages on sites. Spidering issues can be due to:

• The page has not been linked to another page on the site, so will not be found by the spider, as it does not have a link to follow.
• It's written in Macromedia Flash, which is an image based language, rather than a text based language, which the search engine spiders can't read.
• There is a Javascript navigation menu, which search engine robots can have difficulty in penetrating.

These spidering issues are easily dealt with.

• Ensure all of your web pages are linked so that the search engine spiders can crawl from page to page..
• Don't use Macromedia Flash or Javascript, or if you do, ensure that you have an alternative navigation system in the page footer, or a site map, or preferably both.
• Replace the Flash or Javascript navigation menu using Advanced CSS layout techniques. Search engine spiders love CSS Layout!

Search engine spiders are your friends. They ensure that your site is seen by potential customers. To keep them happy, make sure that you feed them links and ensure that you don't try to feed the spiders with Javascript or Flash, as they won't show their appreciation!