Free tools

Robots.txt Examples

Robots.txt file content for medscape.com.

Robot.txt file for: medscape.com

      User-agent: *

Disallow: /viewprogram/*_pnt
Disallow: /druginfo/monoinfobyid*
Disallow: /cp/*
Disallow: /memberprofile/*
Disallow: /noscan/
Disallow: /*/noscan/
Disallow: /viewarticle*_print
Disallow: /viewarticle*_reference
Disallow: /s/
Disallow: /index/*&aspen 
Disallow: /index/*&d2Previous
Disallow: /consult/
Disallow: /mtv/

Sitemap: https://www.medscape.com/sitemap_index.xml

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: / 

User-agent: magpie-crawler
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-Agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: SeekportBot
Crawl-delay: 2