The act of a spider crawling through your website in order to add it to an index.
Due to the ever-changing volume of available data, even the largest search engines are unable to spider all of the internet's pages. As a result, search engines compromise in a number of ways. They may index only home pages for example, or only spider pages that have been submitted to them. Splash Page A splash page provides an initial display for viewing before a searcher can reach the main page. It is similar to a gateway page but is an added step that some consider annoying. Most search engines consider it spam.
for those of us who have arachnophobia (fear of spiders), this web analogy has been carried a bit too far, with people everywhere talking about 'crawlers' and 'spiders'. Yes, Picosearch can do spidering, which really just means that the indexer in your search engine can follow the links on your website to any public web pages, within the parameters you specify. No information is duplicated, (secret stuff is ignored), so the most complete possible index is made for your visitors to search upon. See also indexing.
A program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It's called a spider because it crawls over the Web. Another term for these programs is webcrawler. Because most Web pages contain links to other pages, a spider can start almost anywhere. As soon as it sees a link to another page, it goes off and fetches it. Large search engines, like Alta Vista, have many spiders working in parallel.
The act of requesting and downloading a document by a spider.
While a spider is downloading pages, it is called Spidering. Most modern spiders used by search engines are only responsible for downloading the pages and storing them raw in a temporary database. An indexer is then used to process the page for inclusion in a search engine database. Spiders have a wide range of variables and guidelines that they can be setup to use and follow. Some include: speed at which it downloads pages, whether it will walk or crawl through a website, whether it only goes after index pages, what time of day it is active, which domains it will connect to, how many pages it will accept from one domain.
The process of surfing the web, storing URLs and indexing keywords, links and text. Typically, even the largest search engines cannot spider all of the pages on the net. This is due to the huge amount of data available, the speed at which the new data appears, the use of politeness windows and practical limits on the number of pages that can be visited in a given time . The search engines have to make compromises in order to visit as many sites as possible, and they do this in different ways. For example, some only index the home pages of each site, some only visit sites they're explicitly told about, and some make judgements about the importance of sites (from number and quality of inbound links) before "digging deeper" into the subpages of a site.