Free tools
Robots.txt Examples
Robots.txt file content for drugs.com.
Robot.txt file for: drugs.com
User-agent: Mediapartners-Google User-agent: Twitterbot User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1 Disallow: User-agent: * Disallow: /api/ Disallow: /drug-interactions-all/ Disallow: /interactions-list-drugs.php Disallow: /members_comments_add/ Disallow: /search.php Disallow: /search-wildcard-phonetic.php Disallow: /survey/ User-agent: crawler4j User-agent: Ezooms User-agent: Offline Explorer User-agent: OOZBOT User-agent: oozbot User-agent: SETOOZBOT User-agent: setoozbot User-agent: yacybot Disallow: / Sitemap: https://www.drugs.com/sitemap_index.xml.gz