Free tools

Robots.txt Examples

Robots.txt file content for biomedcentral.com.

Robot.txt file for: biomedcentral.com

      User-agent: *
Disallow: /search
Disallow: */1000$
Disallow: /epdf/
Disallow: tab=
Disallow: /about/institutional-support/membership/
Disallow: /about/membership/members/
Disallow: /about/oa-funding-and-policy-support/
Disallow: /*/*/*/sharedit
Disallow: */platform/contextual*
Disallow: */cas-redirect/*
Disallow: *.ris
Disallow: */articles/*/*/figures/*
Disallow: */articles/*/*/metrics/*
Disallow: */articles/*/*/peer-review/*
Disallow: */articles/*/*/comments/*
Disallow: /placeholder/v1/membership/message

User-Agent: Amazonbot
Disallow: /
User-Agent: anthropic-ai
Disallow: /
User-Agent: Applebot-Extended
Disallow: /
User-Agent: Bytespider
Disallow: /
User-Agent: CCBot
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: GPTBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
User-Agent: GoogleOther
Disallow: /
User-Agent: PerplexityBot
Disallow: /
User-Agent: YouBot
Disallow: /
User-Agent: meta-externalagent
Disallow: /
User-Agent: ImagesiftBot
Disallow: /
User-Agent: cohere-ai
Disallow: /
User-Agent: Diffbot
Disallow: /
User-Agent: FriendlyCrawler
Disallow: /
User-Agent: img2dataset
Disallow: /
User-Agent: omgili
Disallow: /
User-Agent: omgilibot
Disallow: /
User-Agent: Claude-Web
Disallow: /
User-Agent: ChatGPT-User
Disallow: /
Sitemap: https://www.biomedcentral.com/sitemap.xml