Free tools

Robots.txt Examples

Robots.txt file content for bfmtv.com.

Robot.txt file for: bfmtv.com

      User-agent: *
Disallow: /mon-compte/
Disallow: /code-promo/interstitiel/*
Disallow: /bfmtv-et-vous/canal-15-en-15-dates/
Disallow: /bfmtv-et-vous/canal-15-en-15-dates-2/

User-agent: Googlebot-News
Disallow: /static/
Disallow: /services/

User-agent: OrangeBot
Crawl-delay: 1
User-agent: OrangeBot-Collector
Crawl-delay: 1
User-Agent: Slurp
Crawl-delay: 1
User-agent: Adixbot
Crawl-delay: 5
User-Agent: YandexBot
Crawl-delay: 5
User-agent: AhrefsBot
Crawl-delay: 1
User-Agent: Sistrix
Crawl-delay: 5
User-agent: SemrushBot
Crawl-delay: 10
User-agent: Screaming Frog SEO Spider
Crawl-delay: 10
User-agent: SearchmetricsBot
Disallow: /
User-agent: deepcrawl
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Meltawer
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /