Free tools

Robots.txt Examples

Robots.txt file content for ycombinator.com.

Robot.txt file for: ycombinator.com

      User-Agent: *
Disallow: /library?categories=*&*
Allow: /library?categories=*
Disallow: /library?*
Disallow: /companies?*
Allow: /