Free tools

Robots.txt Examples

Robots.txt file content for smh.com.au.

Robot.txt file for: smh.com.au

      User-agent: *
Allow: /
Sitemap: https://www.smh.com.au/sitemaps/news/brands/smh
Sitemap: https://www.smh.com.au/sitemaps/smh-sitemaps-videos.xml
Sitemap: https://www.smh.com.au/sitemaps/smh-navigation-pages.xml
Sitemap: https://www.smh.com.au/sitemaps/smh-sitemaps-articles.xml
Sitemap: https://www.smh.com.au/rss/feed.xml
Sitemap: https://www.theage.com.au/sitemaps/news/brands/theage
Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-videos.xml
Sitemap: https://www.theage.com.au/sitemaps/theage-navigation-pages.xml
Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-articles.xml
Sitemap: https://www.theage.com.au/rss/feed.xml
Sitemap: https://www.brisbanetimes.com.au/sitemaps/news/brands/brisbanetimes
Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-sitemaps-videos.xml
Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-navigation-pages.xml
Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-sitemaps-articles.xml
Sitemap: https://www.brisbanetimes.com.au/rss/feed.xml
Sitemap: https://www.watoday.com.au/sitemaps/news/brands/watoday
Sitemap: https://www.watoday.com.au/sitemaps/watoday-sitemaps-videos.xml
Sitemap: https://www.watoday.com.au/sitemaps/watoday-navigation-pages.xml
Sitemap: https://www.watoday.com.au/sitemaps/watoday-sitemaps-articles.xml
Sitemap: https://www.watoday.com.au/rss/feed.xml
Disallow: /search?text=*