Free tools

Robots.txt Examples

Robots.txt file content for in.gov.

Robot.txt file for: in.gov

      # robots.txt for http://www.IN.gov/

User-agent: *
Disallow: /serv/
Disallow: /cgi-bin/
Disallow: /isdh/drafts_local/
Disallow: /demand
Disallow: /search
Disallow: /ai/errors/
Disallow: /dor/4572.htm
Disallow: /dor/reference/legal/rulings/unused/
Disallow: /dwd/files/swic/
Disallow: /dwd/files/JWIB/
Disallow: /dwd/files/CM_Files/
Disallow: /dwd/files/policy/
Disallow: /dwd/test/ 
Disallow: /ActiveCalendar/mobile/mobilelist.aspx
Disallow: *subscribetocalendar.aspx*
Disallow: *RSSSyndicator.aspx*
Disallow: *downloadtype.aspx*
Disallow: /indot/3212.htm
Disallow: /sos/online_corps/
Disallow: /sos/clerical/
Disallow: /sos/registration/