Free tools

Robots.txt Examples

Robots.txt file content for betterhealth.vic.gov.au.

Robot.txt file for: betterhealth.vic.gov.au

      User-agent: *
Disallow: /js
Disallow: /img
Disallow: /_nuxt
Disallow: /_nuxt/*
User-agent: bingbot
Crawl-delay: 10
User-agent: Baiduspider
Crawl-delay: 10
User-agent: Mediapartners-Google
Crawl-delay: 10
User-agent: www.proximic.com/info/spider.php
Crawl-delay: 10
User-agent: ia_archiver ( crawler@alexa.com)
Crawl-delay: 10
User-agent: GrapeShotCrawler
Crawl-delay: 10
User-agent: msnbot
Crawl-delay: 10
User-agent: YandexBot
Crawl-delay: 10
User-agent: Yahoo! Slurp
Crawl-delay: 30
User-agent: Slurp
Crawl-delay: 30
User-agent: dotbot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: MauiBot
Disallow: /
User-agent: Swiftbot
Crawl-delay: 10
Sitemap: https:/www.betterhealth.vic.gov.au/sitemap.xml