Free tools

Robots.txt Examples

Robots.txt file content for fool.com.

Robot.txt file for: fool.com

      # /robots.txt file for http://www.fool.com/ (prod)
# MauiBot
User-agent: MauiBot
Disallow: /
# ChatGPT
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
# else
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /a/feeds/
Sitemap: https://www.fool.com/sitemap/
Sitemap: https://www.fool.com/wag-sitemap.xml 
Sitemap: https://www.fool.com/money/sitemap.xml
Sitemap: https://www.fool.com/hreflang-tag-sitemap.xml
Disallow: /3910
Disallow: /a/
Disallow: /account/
Disallow: /Account/
Disallow: /admin
Disallow: /Admin
Disallow: /amptest/
Disallow: /art/
Disallow: /Art/
Disallow: /auth/
Disallow: /betaboards/
Disallow: /cdn-cgi/
Disallow: /Common/
Disallow: /dubs/
Disallow: /ecap/
Disallow: /ext-content/
Disallow: /external/
Disallow: /feeds
Disallow: /Feeds
Disallow: /fool/free-report
Disallow: /foolpics
Disallow: /foolPics
Disallow: /FoolPics
Disallow: /free-report
Disallow: /help
Disallow: /Help
Disallow: /includes
Disallow: /Includes
Disallow: /investing/businesswire/
Disallow: /investing/fiercemarkets/
Disallow: /investing/FierceMarkets/
Disallow: /investing/stocks/
Disallow: /investor-alert/
Disallow: /localize
Disallow: /Localize
Disallow: /login/
Disallow: /mailemergency
Disallow: /mailEmergency
Disallow: /MailEmergency
Disallow: /Marketing/EcapSurvey.aspx
Disallow: /mms
Disallow: /news/commentary/
Disallow: /news/xt
Disallow: /News/Xt
Disallow: /News/XT
Disallow: /newsletters/*
Disallow: /nexus/
Disallow: /now50/
Disallow: /offers/
Disallow: /order-results*
Disallow: /order/
Disallow: /p/*
Disallow: /p/60/
Disallow: /partners
Disallow: /Partners
Disallow: /pegulator
Disallow: /Pegulator
Disallow: /premium-reports
Disallow: /private
Disallow: /Private
Disallow: /quote/failed-lookup/
Disallow: /quote/unknown/
Disallow: /reports
Disallow: /scripts
Disallow: /Scripts
Disallow: /search
Disallow: /Search
Disallow: /secure/
Disallow: /server
Disallow: /Server
Disallow: /shop/
Disallow: /Shop/Download/Event/
Disallow: /Shop/OrderResults.aspx
Disallow: /Shop/OrderResults.aspx*
Disallow: /Shop/OrderResults*
Disallow: /snap
Disallow: /Snap
Disallow: /stock-ranker/
Disallow: /test
Disallow: /Test
Disallow: /thank-you/
Disallow: /the-ascent/search/
Disallow: /video-alert
Disallow: /vip/