Free tools
Robots.txt Examples
Robots.txt file content for reuters.com.
Robot.txt file for: reuters.com
# robots.txt for www.reuters.com User-agent: PiplBot Disallow: / User-agent: CCbot Disallow: / # ADO-657700 User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Google-Extended Disallow: / # User-agent: FacebookBot Disallow: / # ADO-658177 User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Timpibot Disallow: / User-agent: Diffbot Disallow: / User-agent: Bytespider Disallow: / User-agent: Claude-Web Disallow: / User-agent: PerplexityBot Disallow: / # ADO-661811 User-agent: Applebot-Extended Disallow: / User-agent: * Disallow: /finance/stocks/option Disallow: /finance/stocks/financialHighlights Disallow: /search Disallow: /site-search/ Disallow: /beta Disallow: /designtech Disallow: /featured-optimize Disallow: /energy-test Disallow: /article/beta Disallow: /sponsored/previewcampaign Disallow: /sponsored/previewarticle Disallow: /test/ Disallow: /commentary Disallow: /news/archive/commentary Disallow: /brandfeatures/venture-capital Disallow: /assets/siteindex Disallow: /article/api/ Disallow: /practical-law-the-journal/search/ Disallow: /info-pages/terms-of-use/custom-services-agreement-23-10/ Disallow: /info-pages/custom-services-agreement-23-10/ Disallow: /info-pages/advertising-guidelines/ Disallow: /pf/api/ Disallow: /fr/ Disallow: /it/ Disallow: /es/ Disallow: /pt/ Disallow: /de/ Disallow: /ar/ SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/?outputType=xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/news-sitemap-index/?outputType=xml SITEMAP: https://www.reuters.com/plus/sitemap-index.xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-plj-index/?outputType=xml SITEMAP: https://www.reuters.com/graphics/sitemap.xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index/pictures/?outputType=xml SITEMAP: https://www.reuters.com/static/video-sitemap/us/sitemap_video_index.xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/topic-sitemap/?outputType=xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/author-sitemap/?outputType=xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/pressrelease-sitemap/?outputType=xml SITEMAP: https://www.reuters.com/arc/outboundfeeds/sitemap-index1/?outputType=xml