Free tools
Robots.txt Examples
Robots.txt file content for in.gov.
Robot.txt file for: in.gov
# robots.txt for http://www.IN.gov/ User-agent: * Disallow: /serv/ Disallow: /cgi-bin/ Disallow: /isdh/drafts_local/ Disallow: /demand Disallow: /search Disallow: /ai/errors/ Disallow: /dor/4572.htm Disallow: /dor/reference/legal/rulings/unused/ Disallow: /dwd/files/swic/ Disallow: /dwd/files/JWIB/ Disallow: /dwd/files/CM_Files/ Disallow: /dwd/files/policy/ Disallow: /dwd/test/ Disallow: /ActiveCalendar/mobile/mobilelist.aspx Disallow: *subscribetocalendar.aspx* Disallow: *RSSSyndicator.aspx* Disallow: *downloadtype.aspx* Disallow: /indot/3212.htm Disallow: /sos/online_corps/ Disallow: /sos/clerical/ Disallow: /sos/registration/