Robots are programs that relatively autonomously fulfil certain tasks in databases, servers or on the internet. Search engines, for instance, use robots to detect and index the contents of web pages for queries.
Robots (or spiders) are automated information retrieval programs collecting data for search engines. There is a standard way of communicating with these robots - intended to control their movements so that the site designer can prevent access to certain areas, for example, directories without html content. As an alternative to the usual "robots.txt" file, you can use meta tags to control the robots. To prevent further indexing of your pages, use the following tag in the page: (The tag is not well supported.)meta name="robots" content="noindex, nofollow" The robots.txt file consists of one or more records separated by one or more blank lines. Each record contains lines of the form "field":"optionalspace""value". The field name is case insensitive. Comments can be added using #. Example: # robots.txt for http://www.codehelp.co.uk Robot: * #any and all are to follow these rules. Disallow: /notthisfolder/ #ignore this folder. Disallow: /northisone/ #ignore this folder too. The robots.txt file needs to be in the main index directory of your site. For more information on robots, email me or search for robots.txt in a search engine.