Free tools
Robots.txt Examples
Robots.txt file content for usmagazine.com.
Robot.txt file for: usmagazine.com
# Sitemap archive Sitemap: https://www.usmagazine.com/sitemap.xml # Video Sitemap Sitemap: https://www.usmagazine.com/feed/video-sitemap/ # Gallery Sitemap Sitemap: https://www.usmagazine.com/gallery-sitemap.xml # Google News Sitemap Sitemap: https://www.usmagazine.com/feed/google-news-sitemap/ # Review Sitemap Sitemap: https://www.usmagazine.com/feed/review-sitemap/ # Authors Sitemap Sitemap: https://www.usmagazine.com/authors-sitemap.xml # Spanish Sitemap: https://www.usmagazine.com/espanol-sitemap.xml Sitemap: https://www.usmagazine.com/espanol-gallery-sitemap.xml Sitemap: https://www.usmagazine.com/espanol-google-news-sitemap.xml User-agent: * Disallow: /*?share= Disallow: /*/dick-van-dyke/celebrities/ Disallow: /*/forever21.com/ Disallow: /sso-dashboard.js Disallow: /article-preview/ Disallow: /wp-json/ Disallow: /_static/ User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ChatGPT Disallow: / User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: YouBot Disallow: / # START YOAST BLOCK # --------------------------- User-agent: * Disallow: # --------------------------- # END YOAST BLOCK