Free tools
Robots.txt Examples
Robots.txt file content for mashable.com.
Robot.txt file for: mashable.com
User-agent: * Disallow: /search Disallow: /archive/ Disallow: /cdn-cgi/ Allow: /*?page=[0-9] User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / Sitemap: https://mashable.com/sitemap-index.xml Sitemap: https://mashable.com/sitemap-news-0.xml