Free tools

Robots.txt Examples

Robots.txt file content for foxnews.com.

Robot.txt file for: foxnews.com

      User-agent: FacebookBot
Disallow: /world/*

User-agent: *
Disallow: /search-results/
Disallow: /video-search/
Disallow: /printer_friendly_story/
Disallow: /printer_friendly_wires/
Disallow: /wires/
Disallow: /xid

Sitemap: https://www.foxnews.com/sitemap.xml
Sitemap: https://www.foxnews.com/sitemap.xml?type=news