Free tools
Robots.txt Examples
Robots.txt file content for va.gov.
Robot.txt file for: va.gov
User-agent: usasearch Allow: / # existing disallow on va.gov (may not be needed) User-agent: Synapse Disallow: / # existing disallow from vets.gov User-Agent: * Disallow: /analytics-opt-out.html Disallow: /cgi-bin/ Disallow: /drupal Disallow: /covid19screen # disallow WIP VAMCs # make sure to add a trailing slash at the end of the path # to prevent sub-directories from being indexed # see https://developers.google.com/search/docs/advanced/robots/create-robots-txt#useful-robots.txt-rules # sitemap index Sitemap: https://www.va.gov/sitemap.xml