Free tools

Robots.txt Examples

Robots.txt file content for hbr.org.

Robot.txt file for: hbr.org

      User-Agent: *
Disallow: /resources/
Allow: /resources/js
Allow: /resources/css
Allow: /resources/images/
Allow: /resources/xml/sitemaps/
Disallow: /fastanswers
Disallow: /my-library*
Disallow: /email-colleague/
Disallow: /add-to-cart/
Disallow: /login*
Disallow: /shopping-cart/
Disallow: /shipping-payment
Disallow: /review-order
Disallow: /order/thank-you/
Disallow: /content/ipad/
Disallow: /newsletters*
Disallow: /product/recommended*
Disallow: /webinar-assessment
Disallow: /search*
Disallow: /academic-subscriptions
Disallow: /alumni-subscriptions
Disallow: /*?*language=es
Disallow: /*?*language=pt
Allow: /resources/html/infographics/2018/ceo-activism-desktop/index.html
Disallow: /resources/html/error/404.html
Disallow: /sponsored/2022/10/disruptors-who-are-changing-their-industries
Disallow: /api/recaptcha/enabled
Disallow: /register?*
Disallow: /resources/css/webscripts.css
User-Agent: 008
Disallow: /
User-Agent: FeedBurner/1.0
Allow: /resources/xml/
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /