Free tools

Robots.txt Examples

Robots.txt file content for ctvnews.ca.

Robot.txt file for: ctvnews.ca

      User-agent: *
Allow: /
Disallow: /search-results/
Disallow: /poll/
Disallow: /logger/
Disallow: /cmlink/
Disallow: /cm/
Disallow: /error/
Disallow: /errorpages/
Disallow: /captcha/
Disallow: /membership/
Disallow: /blogimageupload/
Disallow: /polopolydevelopment/
Disallow: /status/
Disallow: /redirect/
Disallow: /crossdomain.xml
Disallow: /flagcomment/
Disallow: /*stubReferenceName
Disallow: /webdav/
Disallow: /hot-topics/
Disallow: /comments-7.*
Disallow: /*&cache=*
Disallow: /rss/
Disallow: /errors/

Sitemap: http://www.ctvnews.ca/sitemap_news.xml
Sitemap: http://www.ctvnews.ca/sitemap.xml
Sitemap: http://www.ctvnews.ca/sitemap_mobile.xml
Sitemap: http://www.ctvnews.ca/sitemap_video.xml
Sitemap: https://www.ctvnews.ca/shopping-sitemap.xml
Sitemap: https://www.ctvnews.ca/shopping-sitemap-section.xml

User-agent: Google
Disallow: https://election.ctvnews.ca/

User-agent: Twitterbot
Allow: /