Limiting Robots using Robots.txt


So I am trying to build a good robots.txt file that  allows good bots while blocking others.

User-Agent: *
Disallow: /

User-Agent: Googlebot
User-Agent: Googlebot-Mobile
User-Agent: Googlebot-Image
User-Agent: Mediapartners-Google
User-Agent: Adsbot-Google
User-Agent: Slurp
User-Agent: msnbot
User-Agent: msnbot-media
User-agent: Yahoo-MMCrawler
User-Agent: Teoma
Allow: /



Posted on

October 28, 2016

Submit a Comment

Your email address will not be published. Required fields are marked *