Block or Allow Indexing With A Robots.txt File

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. Not every webmaster uses a robots.txt file to instruct search bots as to which files and directories  are or are not allowed to be crawled. But if you do, it is critical the file is properly formatted and configured. Stories abound about disgruntled former consultants or inept webmasters configuring a site's robots.txt file accidently blocking all search bot crawling activity on a site, effectively killing all references to it in the search engine index.

The robots.txt file uses text-based directives written in Robots Exclusion Protocol (REP) to block or allow bot access to specified files, directories, or the entire site. You can specify crawl blocks on one, some, all, or none of the various search engine bots on the Web. You can even configure the crawl rate for bots (if the default bot crawl rate is too aggressive for your servers). Blocking bot access to directories or pages that contain private data, such as scripts, user-specific info, test data, or as-yet-unreleased pages can easily be managed with a robots.txt file. Note, however, that the use of robots.txt is not a security measure. The robots.txt file itself is publicly accessible in the root directory of a website, so anything listed in there to be blocked from indexing is still easily seen by interested parties. Also note that REP directives can be made in the tags at the top of the code in any webpage and in the web server's HTTP Header. There are two important considerations when using /robots.txt file. 1. some robots can and will ignore your robots.txt file especially malware robots that scan the web for security vulnerabilities and email address harvesters used by spammers, and 2. the /robots.txt file is a publicly available file so anyone can see what sections of your server you don't want robots to use.

I advise and lelp all my clients create a robots.txt file even if they don't have any directories that need to be disallowed. After a robots.txt file has been created and uploaded it can be checked to make sure it is correct. More reading on robots.txt files and how to create a file can be read here.