Free tools
Robots.txt Examples
Robots.txt file content for fool.com.
Robot.txt file for: fool.com
# /robots.txt file for http://www.fool.com/ (prod) # MauiBot User-agent: MauiBot Disallow: / # ChatGPT User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / # else User-agent: Bytespider Disallow: / User-agent: ClaudeBot Disallow: / User-agent: * Allow: /a/feeds/ Sitemap: https://www.fool.com/sitemap/ Sitemap: https://www.fool.com/wag-sitemap.xml Sitemap: https://www.fool.com/money/sitemap.xml Sitemap: https://www.fool.com/hreflang-tag-sitemap.xml Disallow: /3910 Disallow: /a/ Disallow: /account/ Disallow: /Account/ Disallow: /admin Disallow: /Admin Disallow: /amptest/ Disallow: /art/ Disallow: /Art/ Disallow: /auth/ Disallow: /betaboards/ Disallow: /cdn-cgi/ Disallow: /Common/ Disallow: /dubs/ Disallow: /ecap/ Disallow: /ext-content/ Disallow: /external/ Disallow: /feeds Disallow: /Feeds Disallow: /fool/free-report Disallow: /foolpics Disallow: /foolPics Disallow: /FoolPics Disallow: /free-report Disallow: /help Disallow: /Help Disallow: /includes Disallow: /Includes Disallow: /investing/businesswire/ Disallow: /investing/fiercemarkets/ Disallow: /investing/FierceMarkets/ Disallow: /investing/stocks/ Disallow: /investor-alert/ Disallow: /localize Disallow: /Localize Disallow: /login/ Disallow: /mailemergency Disallow: /mailEmergency Disallow: /MailEmergency Disallow: /Marketing/EcapSurvey.aspx Disallow: /mms Disallow: /news/commentary/ Disallow: /news/xt Disallow: /News/Xt Disallow: /News/XT Disallow: /newsletters/* Disallow: /nexus/ Disallow: /now50/ Disallow: /offers/ Disallow: /order-results* Disallow: /order/ Disallow: /p/* Disallow: /p/60/ Disallow: /partners Disallow: /Partners Disallow: /pegulator Disallow: /Pegulator Disallow: /premium-reports Disallow: /private Disallow: /Private Disallow: /quote/failed-lookup/ Disallow: /quote/unknown/ Disallow: /reports Disallow: /scripts Disallow: /Scripts Disallow: /search Disallow: /Search Disallow: /secure/ Disallow: /server Disallow: /Server Disallow: /shop/ Disallow: /Shop/Download/Event/ Disallow: /Shop/OrderResults.aspx Disallow: /Shop/OrderResults.aspx* Disallow: /Shop/OrderResults* Disallow: /snap Disallow: /Snap Disallow: /stock-ranker/ Disallow: /test Disallow: /Test Disallow: /thank-you/ Disallow: /the-ascent/search/ Disallow: /video-alert Disallow: /vip/