Free tools

Robots.txt Examples

Robots.txt file content for merckmanuals.com.

Robot.txt file for: merckmanuals.com

      User-agent: *
Crawl-delay: 5
Disallow: */sitecore
Disallow: */Sitecore
Disallow: */custom/
Disallow: */Custom/
Disallow: */news/external/
Disallow: */News/External/
Disallow: */pagerevalidation
Disallow: */multimedia/zk/
Disallow: */Multimedia/zk/
Disallow: */multimedia/Zk/
Disallow: */Multimedia/Zk/
Disallow: */multimedia/ZK/
Disallow: */Multimedia/ZK/
Disallow: */professional/drug-names-generic-and-brand
Disallow: */home/drug-names-generic-and-brand
Disallow: */home/monograph/*
Disallow: */professional/monograph/*
User-agent: Googlebot-Image
Disallow: *blausen*
#Baiduspider
User-agent: Baiduspider
Disallow: /zh/

Sitemap: https://www.merckmanuals.com/en-ca/sitemap.ashx
Sitemap: https://www.merckmanuals.com/sitemap.ashx
Sitemap: https://www.merckmanuals.com/es-us/sitemap.ashx
Sitemap: https://www.merckmanuals.com/fr-ca/sitemap.ashx
User-agent: Googlebot-Image
Disallow: *blausen*