Free tools

Robots.txt Examples

Robots.txt file content for aljazeera.com.

Robot.txt file for: aljazeera.com

      User-agent: *
Sitemap: https://www.aljazeera.com/sitemap.xml
Sitemap: https://www.aljazeera.com/sitemaps/article-archive.xml
Sitemap: https://www.aljazeera.com/sitemaps/article-new.xml
Sitemap: https://www.aljazeera.com/sitemaps/video-archive.xml
Sitemap: https://www.aljazeera.com/sitemaps/video-new.xml
Disallow: /api
Disallow: /asset-manifest.json
Allow: /search/$
Disallow: /search/
Disallow: /home/search?q=