Free tools

Robots.txt Examples

Robots.txt file content for ca.gov.

Robot.txt file for: ca.gov

      User-agent: *
Allow: /
Disallow: /ads.txt$
Disallow: /app-ads.txt$

Sitemap: https://www.ca.gov/sitemap.xml