Free tools

Robots.txt Examples

Robots.txt file content for bustle.com.

Robot.txt file for: bustle.com

      User-agent: *
Disallow: 
Disallow: /search?*

User-agent: Amazonbot
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: NewsWhipbot
Disallow: /

Sitemap: https://www.bustle.com/sitemaps/pages.xml
Sitemap: https://www.bustle.com/sitemaps/news.xml