Free tools

Robots.txt Examples

Robots.txt file content for nj.gov.

Robot.txt file for: nj.gov

      User-agent: *
Disallow: /cgi-bin/homelandsecurity/
Disallow: /cgi-bin/dobi/licenseesearch/
Disallow: /cgi-bin/consumeraffairs/search/
Disallow: /cgi-bin/state/
Disallow: /Support/
Disallow: /treasury/treasdocuments/
Disallow: /highereducation/higheddocs/