RewriteEngine On
RewriteCond %{http_user_agent} !(googlebot|Msnbot|Slurp) [NC]
RewriteRule ^robots\.txt$ http://mysite.com/ [R,NE,L]
AddHandler application/x-httpd-php .txt
So if you are not one of the three big search engines and want to reach
the robots.txt file you will be redirected to the main domain.
The last row makes your robots.txt file behave as a php script.
2. Open a text editor and insert the code below into a file that you
save as reversedns.php and upload it to your root folder.
If the robot can not be validated the script will redirect it to the main domain.
If somebody masks as googlebot he will fail during this valadiation.
3. Open the robots.txt file you would like to protect and insert the code below to the first line.
PHP Code:
Original post found at http://www.ukwebmasterworld.com/programming/7256-hide-your-robots-txt-visitors-show-only-validated-robots.html and copied just in case it goes offline
