Free tools
Robots.txt Examples
Robots.txt file content for apnews.com.
Robot.txt file for: apnews.com
User-Agent: * Disallow: Disallow: *_ptid=* Disallow: *?prx_t=* Disallow: /press-release/* Disallow: /blaize* Disallow: /zephr* Disallow: /search?q=* Disallow: /*.rss Disallow: /buyline-shopping/* Disallow: /buyline-personal-finance/* User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / Sitemap: https://apnews.com/ap-sitemap.xml Sitemap: https://apnews.com/news-sitemap.xml Sitemap: https://apnews.com/video-sitemap.xml # Disallow Rules User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot Disallow: /