Free tools
Robots.txt Examples
Robots.txt file content for medicalnewstoday.com.
Robot.txt file for: medicalnewstoday.com
User-agent: Mediapartners-Google* Disallow: User-agent: Nutch Crawl-delay: 5 Disallow: User-agent: Slurp Disallow: /*.gif$ Disallow: /*.jpg$ User-agent: * Crawl-delay: 5 Disallow: /linkfwd.php Disallow: /counters.php # Wordpress Previews Disallow: /articles/mnt-* Disallow: /program/mnt-* # API Routes Disallow: /api/* # Invalid URLs Disallow: */null$ Disallow: */inline$ User-agent: GPTBot Disallow: / # Sitemaps Sitemap: https://www.medicalnewstoday.com/sitemap.xml # Widget Sampler Disallow: /articles/widget-sampler # Static Test Articles Disallow: /test/