User-agent: * Allow: / Disallow: /externalLink.asp Disallow: /externallink.asp Disallow: /feed/ Crawl-delay: 5 User-Agent: rogerbot Disallow: /externalLink.asp Disallow: /externallink.asp Disallow: /search/ Disallow: /thread/ Disallow: /forum/ Disallow: /feed/ Allow: / Crawl-delay: 5 User-Agent: dotbot Disallow: /externalLink.asp Disallow: /externallink.asp Disallow: /search/ Disallow: /thread/ Disallow: /forum/ Disallow: /feed/ Allow: / Crawl-delay: 5