Free tools
Robots.txt Examples
Robots.txt file content for smh.com.au.
Robot.txt file for: smh.com.au
User-agent: * Allow: / Sitemap: https://www.smh.com.au/sitemaps/news/brands/smh Sitemap: https://www.smh.com.au/sitemaps/smh-sitemaps-videos.xml Sitemap: https://www.smh.com.au/sitemaps/smh-navigation-pages.xml Sitemap: https://www.smh.com.au/sitemaps/smh-sitemaps-articles.xml Sitemap: https://www.smh.com.au/rss/feed.xml Sitemap: https://www.theage.com.au/sitemaps/news/brands/theage Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-videos.xml Sitemap: https://www.theage.com.au/sitemaps/theage-navigation-pages.xml Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-articles.xml Sitemap: https://www.theage.com.au/rss/feed.xml Sitemap: https://www.brisbanetimes.com.au/sitemaps/news/brands/brisbanetimes Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-sitemaps-videos.xml Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-navigation-pages.xml Sitemap: https://www.brisbanetimes.com.au/sitemaps/brisbanetimes-sitemaps-articles.xml Sitemap: https://www.brisbanetimes.com.au/rss/feed.xml Sitemap: https://www.watoday.com.au/sitemaps/news/brands/watoday Sitemap: https://www.watoday.com.au/sitemaps/watoday-sitemaps-videos.xml Sitemap: https://www.watoday.com.au/sitemaps/watoday-navigation-pages.xml Sitemap: https://www.watoday.com.au/sitemaps/watoday-sitemaps-articles.xml Sitemap: https://www.watoday.com.au/rss/feed.xml Disallow: /search?text=*