Free tools
Robots.txt Examples
Robots.txt file content for medscape.com.
Robot.txt file for: medscape.com
User-agent: * Disallow: /viewprogram/*_pnt Disallow: /druginfo/monoinfobyid* Disallow: /cp/* Disallow: /memberprofile/* Disallow: /noscan/ Disallow: /*/noscan/ Disallow: /viewarticle*_print Disallow: /viewarticle*_reference Disallow: /s/ Disallow: /index/*&aspen Disallow: /index/*&d2Previous Disallow: /consult/ Disallow: /mtv/ Sitemap: https://www.medscape.com/sitemap_index.xml User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: ChatGPT-User Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: SeekportBot Crawl-delay: 2