Free tools

Robots.txt Examples

Robots.txt file content for smithsonianmag.com.

Robot.txt file for: smithsonianmag.com

      User-agent: *
Disallow: /accounts/*
Disallow: /museumdays/accounts/
Disallow: /accounts/login/
Disallow: /search?
Disallow: /rss?
Disallow: /article/preview/*/
Disallow: /blogs/blogpost/preview/*
Disallow: /dashboard/
Disallow: /acoustic/

Sitemap: https://www.smithsonianmag.com/sitemap.xml