Free tools

Robots.txt Examples

Robots.txt file content for standard.co.uk.

Robot.txt file for: standard.co.uk

      
User-agent: *
Disallow: /api/
Disallow: /internal-api/
Disallow: /*ILC-refresh

Sitemap: https://www.standard.co.uk/sitemaps/googlenews
Sitemap: https://www.standard.co.uk/sitemap.xml

User-agent: Nutch
Disallow: /