Free tools

Robots.txt Examples

Robots.txt file content for apnews.com.

Robot.txt file for: apnews.com

      User-Agent: *
Disallow:
Disallow: *_ptid=*
Disallow: *?prx_t=*
Disallow: /press-release/*
Disallow: /blaize*
Disallow: /zephr*
Disallow: /search?q=*
Disallow: /*.rss
Disallow: /buyline-shopping/*
Disallow: /buyline-personal-finance/*

User-agent: CCBot
Disallow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://apnews.com/ap-sitemap.xml
Sitemap: https://apnews.com/news-sitemap.xml
Sitemap: https://apnews.com/video-sitemap.xml

# Disallow Rules
User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /