Free tools

Robots.txt Examples

Robots.txt file content for apnews.com.

Robot.txt file for: apnews.com

      User-Agent: *
Disallow:
Disallow: *_ptid=*
Disallow: *?prx_t=*
Disallow: /press-release/*
Disallow: /blaize*
Disallow: /zephr*
Disallow: /search?q=*

User-agent: CCBot
Disallow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://apnews.com/ap-sitemap.xml
Sitemap: https://apnews.com/news-sitemap-content.xml
Sitemap: https://apnews.com/video-sitemap.xml
Sitemap: https://apnews.com/buyline-personal-finance/sitemap.xml
Sitemap: https://apnews.com/buyline-shopping/sitemap.xml

# Disallow Rules
User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /