Free tools
Robots.txt Examples
Robots.txt file content for aljazeera.com.
Robot.txt file for: aljazeera.com
User-agent: * Sitemap: https://www.aljazeera.com/sitemap.xml Sitemap: https://www.aljazeera.com/sitemaps/article-archive.xml Sitemap: https://www.aljazeera.com/sitemaps/article-new.xml Sitemap: https://www.aljazeera.com/sitemaps/video-archive.xml Sitemap: https://www.aljazeera.com/sitemaps/video-new.xml Disallow: /api Disallow: /asset-manifest.json Allow: /search/$ Disallow: /search/ Disallow: /home/search?q=