Free tools

Robots.txt Examples

Robots.txt file content for loc.gov.

Robot.txt file for: loc.gov

      
User-agent: 008
Disallow: /

#Baiduspider
User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /web_arch/
Disallow: /rr/mopic/staff
Disallow: /loc/volunteers
Disallow: /ficmanagers
Disallow: /preserv/extranet/
Disallow: /myloc
Disallow: /nationalfilmregistry
Disallow: /fedsearch
Disallow: /search
Disallow: /pictures/search
Disallow: /pictures/related
Crawl-Delay: 5

Sitemap: https://www.loc.gov/sitemap.xml