Free tools

Robots.txt Examples

Robots.txt file content for mashable.com.

Robot.txt file for: mashable.com

      User-agent: *
Disallow: /search
Disallow: /archive/
Disallow: /cdn-cgi/
Allow: /*?page=[0-9]

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

Sitemap: https://mashable.com/sitemap-index.xml
Sitemap: https://mashable.com/sitemap-news-0.xml