Free tools
Robots.txt Examples
Robots.txt file content for hbr.org.
Robot.txt file for: hbr.org
User-Agent: * Disallow: /resources/ Allow: /resources/js Allow: /resources/css Allow: /resources/images/ Allow: /resources/xml/sitemaps/ Disallow: /fastanswers Disallow: /my-library* Disallow: /email-colleague/ Disallow: /add-to-cart/ Disallow: /login* Disallow: /shopping-cart/ Disallow: /shipping-payment Disallow: /review-order Disallow: /order/thank-you/ Disallow: /content/ipad/ Disallow: /newsletters* Disallow: /product/recommended* Disallow: /webinar-assessment Disallow: /search* Disallow: /academic-subscriptions Disallow: /alumni-subscriptions Disallow: /*?*language=es Disallow: /*?*language=pt Allow: /resources/html/infographics/2018/ceo-activism-desktop/index.html Disallow: /resources/html/error/404.html Disallow: /sponsored/2022/10/disruptors-who-are-changing-their-industries Disallow: /api/recaptcha/enabled Disallow: /register?* Disallow: /resources/css/webscripts.css User-Agent: 008 Disallow: / User-Agent: FeedBurner/1.0 Allow: /resources/xml/ User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: /