Robots.txt for state.gov
textile
crawl-delay: 5
# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:
Sitemap: https://www.state.gov/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK
textile
crawl-delay: 5
# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:
Sitemap: https://www.state.gov/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK
Don't have an account? We will create one for you.
Enter the OTP send to
in seconds.