Free tools
Robots.txt Examples
Robots.txt file content for biomedcentral.com.
Robot.txt file for: biomedcentral.com
User-agent: * Disallow: /search Disallow: */1000$ Disallow: /epdf/ Disallow: tab= Disallow: /about/institutional-support/membership/ Disallow: /about/membership/members/ Disallow: /about/oa-funding-and-policy-support/ Disallow: /*/*/*/sharedit Disallow: */platform/contextual* Disallow: */cas-redirect/* Disallow: *.ris Disallow: */articles/*/*/figures/* Disallow: */articles/*/*/metrics/* Disallow: */articles/*/*/peer-review/* Disallow: */articles/*/*/comments/* Disallow: /placeholder/v1/membership/message User-Agent: Amazonbot Disallow: / User-Agent: anthropic-ai Disallow: / User-Agent: Applebot-Extended Disallow: / User-Agent: Bytespider Disallow: / User-Agent: CCBot Disallow: / User-Agent: ClaudeBot Disallow: / User-Agent: FacebookBot Disallow: / User-Agent: GPTBot Disallow: / User-Agent: Google-Extended Disallow: / User-Agent: GoogleOther Disallow: / User-Agent: PerplexityBot Disallow: / User-Agent: YouBot Disallow: / User-Agent: meta-externalagent Disallow: / User-Agent: ImagesiftBot Disallow: / User-Agent: cohere-ai Disallow: / User-Agent: Diffbot Disallow: / User-Agent: FriendlyCrawler Disallow: / User-Agent: img2dataset Disallow: / User-Agent: omgili Disallow: / User-Agent: omgilibot Disallow: / User-Agent: Claude-Web Disallow: / User-Agent: ChatGPT-User Disallow: / Sitemap: https://www.biomedcentral.com/sitemap.xml