Free tools

Robots.txt Examples

Robots.txt file content for craigslist.org.

Robot.txt file for: craigslist.org

      User-agent: *
Disallow: /reply
Disallow: /fb/
Disallow: /suggest
Disallow: /flag
Disallow: /mf
Disallow: /mailflag
Disallow: /eaf

User-agent: claudebot
Disallow: /