Free tools
Robots.txt Examples
Robots.txt file content for teenvogue.com.
Robot.txt file for: teenvogue.com
User-agent: * Disallow: /user/ Disallow: /search Disallow: /search/ Disallow: /*.jsp$ Disallow: /contests/ Disallow: /preview/ Disallow: /photos/ Disallow: /video/right_rail* Disallow: /*?redirectURL= Disallow: /*{{url}} Disallow: /*%7B%7Burl%7D%7D Disallow: /*{{section}} Disallow: /*%7B%7Bsection%7D%7D Disallow: /auth/* Disallow: /account/* Disallow: /categories/*/revisions Disallow: /categories/*/rels Disallow: /categories/*/archive Disallow: /categories/*/identifiers Disallow: /articles/*/revisions Disallow: /articles/*/publish Disallow: /articles/*/next Disallow: /articles/*/previous Disallow: /articles/*/rels Disallow: /articles/*/archive Disallow: /articles/*/identifiers Disallow: /contributor/*/revisions Disallow: /contributor/*/publish Disallow: /contributor/*/next Disallow: /contributor/*/previous Disallow: /contributor/*/rels Disallow: /contributor/*/archive Disallow: /contributor/*/identifiers Disallow: /cnevideos/*/revisions Disallow: /cnevideos/*/rels Disallow: /cnevideos/*/archive Disallow: /articles/published/more Disallow: /contributor/published/more Disallow: /galleries/ Disallow: /product/ Disallow: /user-context?referrer Disallow: /services.min.js?_= Disallow: /com.condenast/yv8* User-agent: Google-InspectionTool Allow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: omgili Disallow: / User-agent: PerplexityBot Disallow: / User-agent: YouBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: Applebot-Extended Disallow: / Sitemap: https://www.teenvogue.com/sitemap.xml Sitemap: https://www.teenvogue.com/branded-sitemap.xml Sitemap: https://www.teenvogue.com/feed/google-latest-news/sitemap-google-news Sitemap: https://www.teenvogue.com/video/sitemap.xml Sitemap: https://www.teenvogue.com/categories-sitemap.xml Sitemap: https://www.teenvogue.com/contributors-sitemap.xml Sitemap: https://www.teenvogue.com/branded-sitemap.xml Sitemap: https://www.teenvogue.com/sitemap-archive-1.xml Sitemap: https://www.teenvogue.com/bundles-sitemap.xml