Free tools

Robots.txt Examples

Robots.txt file content for nps.gov.

Robot.txt file for: nps.gov

      User-agent: *
Disallow: /ns/
Disallow: /search/
Disallow: /loader.cfm
Disallow: /*loader.cfm*
User-agent: bingbot
Crawl-delay: 5