Free tools

Robots.txt Examples

Robots.txt file content for clarin.com.

Robot.txt file for: clarin.com

      User-agent: *
Disallow: /api/
Disallow: /redaccionfiler/*
Disallow: /viafoura/*
Disallow: /autor/*?
Disallow: /videos/*?
Disallow: /videoloop/*_3_

Sitemap: https://www.clarin.com/sitemaps/index.xml
Sitemap: https://www.clarin.com/sitemaps/index_services.xml
Sitemap: https://www.clarin.com/sitemaps/sitemap_google_news.xml