Free tools
Robots.txt Examples
Robots.txt file content for cnn.com.
Robot.txt file for: cnn.com
Sitemap: https://www.cnn.com/sitemaps/cnn/index.xml Sitemap: https://www.cnn.com/sitemaps/cnn/news.xml Sitemap: https://www.cnn.com/sitemap/news.xml Sitemap: https://www.cnn.com/sitemaps/sitemap-section.xml Sitemap: https://www.cnn.com/sitemaps/sitemap-interactive.xml Sitemap: https://www.cnn.com/ampstories/sitemap.xml Sitemap: https://edition.cnn.com/sitemaps/news.xml Sitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap.xml Sitemap: https://www.cnn.com/cnn-underscored/money/sitemap-news.xml Sitemap: https://www.cnn.com/sitemap/section/politics.xml Sitemap: https://www.cnn.com/sitemap/article/opinions.xml Sitemap: https://www.cnn.com/sitemap/article.xml Sitemap: https://www.cnn.com/sitemap/section.xml Sitemap: https://www.cnn.com/sitemap/video.xml Sitemap: https://www.cnn.com/sitemap/gallery.xml Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml Sitemap: https://www.cnn.com/sitemap/live-story.xml Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml User-agent: anthropic-ai User-agent: AwarioRssBot User-agent: AwarioSmartBot User-agent: Bytespider User-agent: CCBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-Web User-agent: cohere-ai User-agent: DataForSeoBot User-agent: Diffbot User-agent: FacebookBot User-agent: GPTBot User-agent: Google-Extended User-agent: magpie-crawler User-agent: NewsNow User-agent: news-please User-agent: omgili User-agent: omgilibot User-agent: PerplexityBot User-agent: PiplBot User-agent: Scrapy User-agent: TurnitinBot Disallow: / User-agent: * Allow: /partners/ipad/live-video.json Disallow: /*.jsx$ Disallow: *.jsx$ Disallow: /*.jsx/ Disallow: *.jsx? Disallow: /ads/ Disallow: /aol/ Disallow: /api/ Disallow: /beta/ Disallow: /browsers/ Disallow: /cl/ Disallow: /cnews/ Disallow: /cnn_adspaces Disallow: /cnnbeta/ Disallow: /cnnintl_adspaces Disallow: /development Disallow: /editionssi Disallow: /help/cnnx.html Disallow: /NewsPass Disallow: /NOKIA Disallow: /partners/ Disallow: /pipeline/ Disallow: /pointroll/ Disallow: /POLLSERVER/ Disallow: /pr/ Disallow: /privacy Disallow: /PV/ Disallow: /Quickcast/ Disallow: /quickcast/ Disallow: /QUICKNEWS/ Disallow: /search Disallow: /subscriptions/video-docs Disallow: /terms Disallow: /test/ Disallow: /virtual/ Disallow: /WEB-INF/ Disallow: /web.projects/ Disallow: /webview/ Disallow: /games User-agent: Googlebot-News Disallow: /sponsor