Free tools
Robots.txt Examples
Robots.txt file content for ctvnews.ca.
Robot.txt file for: ctvnews.ca
User-agent: * Allow: / Disallow: /search-results/ Disallow: /poll/ Disallow: /logger/ Disallow: /cmlink/ Disallow: /cm/ Disallow: /error/ Disallow: /errorpages/ Disallow: /captcha/ Disallow: /membership/ Disallow: /blogimageupload/ Disallow: /polopolydevelopment/ Disallow: /status/ Disallow: /redirect/ Disallow: /crossdomain.xml Disallow: /flagcomment/ Disallow: /*stubReferenceName Disallow: /webdav/ Disallow: /hot-topics/ Disallow: /comments-7.* Disallow: /*&cache=* Disallow: /rss/ Disallow: /errors/ Sitemap: http://www.ctvnews.ca/sitemap_news.xml Sitemap: http://www.ctvnews.ca/sitemap.xml Sitemap: http://www.ctvnews.ca/sitemap_mobile.xml Sitemap: http://www.ctvnews.ca/sitemap_video.xml Sitemap: https://www.ctvnews.ca/shopping-sitemap.xml Sitemap: https://www.ctvnews.ca/shopping-sitemap-section.xml User-agent: Google Disallow: https://election.ctvnews.ca/ User-agent: Twitterbot Allow: /