Definitions for "Robots.txt"
Keywords:  txt, spider, obey, exclusion, crawl
A file that gives instructions to most spiders as to which pages on a site they are allowed to look at and index. Complete explanation of the Robots Exclusion Protocol Here.
A text file kept in the root of a website. It tells robots, spiders and crawlers where they are allowed to go and not allowed to go on a website.
This is a file which well-behave spiders read to determine which parts of a website they may visit.