Free tools

Robots.txt Examples

Robots.txt file content for virginia.gov.

Robot.txt file for: virginia.gov

      User-agent: *
Disallow: /site-config
Disallow: /_sidebar
Disallow: /_accordion
Disallow: /testing
Disallow: /archive
Disallow: /demo
Disallow: /trash
Disallow: /sidebar
Disallow: /accordion
Disallow: /modules
Disallow: /_*


User-agent: *
Disallow: /search
Disallow: /404
Disallow: /home-features
Disallow: /agencies/*/services
Sitemap: https://www.virginia.gov/sitemap.xml