November 26, 2024

Robots.txt for state.gov

textile

crawl-delay: 5
# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://www.state.gov/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK