Free tools

Robots.txt Examples

Robots.txt file content for usmagazine.com.

Robot.txt file for: usmagazine.com

      # Sitemap archive
Sitemap: https://www.usmagazine.com/sitemap.xml

# Video Sitemap
Sitemap: https://www.usmagazine.com/feed/video-sitemap/
# Gallery Sitemap
Sitemap: https://www.usmagazine.com/feed/gallery-sitemap/
# Google News Sitemap
Sitemap: https://www.usmagazine.com/feed/google-news-sitemap/
# Review Sitemap
Sitemap: https://www.usmagazine.com/feed/review-sitemap/
# Authors Sitemap
Sitemap: https://www.usmagazine.com/authors-sitemap.xml

User-agent: *
Disallow: /search-results/
Disallow: /search*
Disallow: /*?_s=
Disallow: /*?share=
Disallow: /*/dick-van-dyke/celebrities/
Disallow: /*/forever21.com/
Disallow: /sso-dashboard.js
Disallow: /article-preview/
Disallow: /wp-json/
Disallow: /_static/

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

# ---------------------------
# END YOAST BLOCK