Free tools

Robots.txt Examples

Robots.txt file content for nasa.gov.

Robot.txt file for: nasa.gov

      User-agent: usasearch
Crawl-delay: 1
Allow: /

User-agent: *
Allow: /