So I am trying to build a good robots.txt file that allows good bots while blocking others.
User-Agent: * Disallow: / User-Agent: Googlebot User-Agent: Googlebot-Mobile User-Agent: Googlebot-Image User-Agent: Mediapartners-Google User-Agent: Adsbot-Google User-Agent: Slurp User-Agent: msnbot User-Agent: msnbot-media User-agent: Yahoo-MMCrawler User-Agent: Teoma Allow: /