Don't be afraid of the spiders! The main spidering search engines, Google, Yahoo and Bing, use programs called search engine robots or search engine spiders to find web pages to build their index. (You may also hear them called, index robots, search spiders, indexing robots and various combinations of these terms). The search engine spiders are employed 24 hours a day, 7 days a week, to ensure that their index is kept up to date.
The search engine spiders are not limited to a pre-defined list of web pages to look at and to index, they simply follow links on each page, that they index, and then they follow the links on those pages, etc. The process of following links is called spidering .
Search Engine Spiders Index as they go
The search engine robots come onto your index page from a link from another site. It stores all the information on your index page and then follows the links on that page and then indexes those, then follows the links on those pages, and so on. That's how the search engines builds its index of millions of web pages.
Search engine spiders sometimes have problems finding pages on sites. Spidering issues can be due to:
- The page has not been linked to another page on the site, so will not be found by the spider, as it does not have a link to follow.
- It's written in Adobe Flash, which is an image based language, rather than a text based language, which the search engine spiders can't read.
These spidering issues are easily dealt with.
- Ensure all of your web pages are linked so that the search engine spiders can crawl from page to page..