Free tools
Robots.txt Examples
Robots.txt file content for merckmanuals.com.
Robot.txt file for: merckmanuals.com
User-agent: * Crawl-delay: 5 Disallow: */sitecore Disallow: */Sitecore Disallow: */custom/ Disallow: */Custom/ Disallow: */news/external/ Disallow: */News/External/ Disallow: */pagerevalidation Disallow: */multimedia/zk/ Disallow: */Multimedia/zk/ Disallow: */multimedia/Zk/ Disallow: */Multimedia/Zk/ Disallow: */multimedia/ZK/ Disallow: */Multimedia/ZK/ Disallow: */professional/drug-names-generic-and-brand Disallow: */home/drug-names-generic-and-brand Disallow: */home/monograph/* Disallow: */professional/monograph/* User-agent: Googlebot-Image Disallow: *blausen* #Baiduspider User-agent: Baiduspider Disallow: /zh/ Sitemap: https://www.merckmanuals.com/en-ca/sitemap.ashx Sitemap: https://www.merckmanuals.com/sitemap.ashx Sitemap: https://www.merckmanuals.com/es-us/sitemap.ashx Sitemap: https://www.merckmanuals.com/fr-ca/sitemap.ashx User-agent: Googlebot-Image Disallow: *blausen*