Free tools
Robots.txt Examples
Robots.txt file content for betterhealth.vic.gov.au.
Robot.txt file for: betterhealth.vic.gov.au
User-agent: * Disallow: /js Disallow: /img Disallow: /_nuxt Disallow: /_nuxt/* User-agent: bingbot Crawl-delay: 10 User-agent: Baiduspider Crawl-delay: 10 User-agent: Mediapartners-Google Crawl-delay: 10 User-agent: www.proximic.com/info/spider.php Crawl-delay: 10 User-agent: ia_archiver ( crawler@alexa.com) Crawl-delay: 10 User-agent: GrapeShotCrawler Crawl-delay: 10 User-agent: msnbot Crawl-delay: 10 User-agent: YandexBot Crawl-delay: 10 User-agent: Yahoo! Slurp Crawl-delay: 30 User-agent: Slurp Crawl-delay: 30 User-agent: dotbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MauiBot Disallow: / User-agent: Swiftbot Crawl-delay: 10 Sitemap: https:/www.betterhealth.vic.gov.au/sitemap.xml