Free tools
Robots.txt Examples
Robots.txt file content for craigslist.org.
Robot.txt file for: craigslist.org
User-agent: * Disallow: /reply Disallow: /fb/ Disallow: /suggest Disallow: /flag Disallow: /mf Disallow: /mailflag Disallow: /eaf User-agent: claudebot Disallow: /