Free tools

Robots.txt Examples

Robots.txt file content for wikiwand.com.

Robot.txt file for: wikiwand.com

      User-agent: GPTBot
Disallow: /

User-agent: Bytedance
User-agent: Bytespider
Disallow: /

User-agent: SemrushBot
User-agent: SiteAuditBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SplitSignalBot
User-agent: SemrushBot-OCOB
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: filterdb.iss.net\/crawler
Disallow: /

User-agent: *
Disallow: */w/index.php*