Free tools
Robots.txt Examples
Robots.txt file content for bfmtv.com.
Robot.txt file for: bfmtv.com
User-agent: * Disallow: /mon-compte/ Disallow: /code-promo/interstitiel/* Disallow: /bfmtv-et-vous/canal-15-en-15-dates/ Disallow: /bfmtv-et-vous/canal-15-en-15-dates-2/ User-agent: Googlebot-News Disallow: /static/ Disallow: /services/ User-agent: OrangeBot Crawl-delay: 1 User-agent: OrangeBot-Collector Crawl-delay: 1 User-Agent: Slurp Crawl-delay: 1 User-agent: Adixbot Crawl-delay: 5 User-Agent: YandexBot Crawl-delay: 5 User-agent: AhrefsBot Crawl-delay: 1 User-Agent: Sistrix Crawl-delay: 5 User-agent: SemrushBot Crawl-delay: 10 User-agent: Screaming Frog SEO Spider Crawl-delay: 10 User-agent: SearchmetricsBot Disallow: / User-agent: deepcrawl Disallow: / User-agent: rogerbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Meltawer Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: /