Free tools
Robots.txt Examples
Robots.txt file content for smithsonianmag.com.
Robot.txt file for: smithsonianmag.com
User-agent: * Disallow: /accounts/* Disallow: /museumdays/accounts/ Disallow: /accounts/login/ Disallow: /search? Disallow: /rss? Disallow: /article/preview/*/ Disallow: /blogs/blogpost/preview/* Disallow: /dashboard/ Disallow: /acoustic/ Sitemap: https://www.smithsonianmag.com/sitemap.xml