Free tools

Robots.txt Examples

Robots.txt file content for va.gov.

Robot.txt file for: va.gov

      User-agent: usasearch
Allow: /

# existing disallow on va.gov (may not be needed)
User-agent: Synapse
Disallow: /

# existing disallow from vets.gov
User-Agent: *
Disallow: /analytics-opt-out.html
Disallow: /cgi-bin/
Disallow: /drupal
Disallow: /covid19screen

# disallow WIP VAMCs
# make sure to add a trailing slash at the end of the path
# to prevent sub-directories from being indexed
# see https://developers.google.com/search/docs/advanced/robots/create-robots-txt#useful-robots.txt-rules


# sitemap index
Sitemap: https://www.va.gov/sitemap.xml