Free tools
Robots.txt Examples
Robots.txt file content for cbsnews.com.
Robot.txt file for: cbsnews.com
# www.robotstxt.org/ # www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 User-agent: * Disallow: */search/* Disallow: /index.php/* Disallow: /election-results-data/* Disallow: /common Disallow: /htdocs/* Disallow: /user/login* Disallow: /app/?* # PER CBS-N ENG FINAL ROUTES DOC Disallow: /1318 Disallow: /1319 Disallow: /1328 Disallow: /1344 Disallow: /1770 Disallow: /2240 Disallow: /2303 Disallow: /2304 Disallow: /2305 Disallow: /2306 Disallow: /2307 Disallow: /2308 Disallow: /2332 Disallow: /2741 Disallow: /2994 Disallow: /8324 Disallow: /8601 Disallow: /8614 Disallow: /8618 Disallow: /9742 Disallow: /9744 Sitemap: https://www.cbsnews.com/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/baltimore/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/boston/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/chicago/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/colorado/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/gooddaysacramento/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/detroit/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/texas/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/losangeles/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/cbslocal/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/miami/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/minnesota/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/newyork/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/philadelphia/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/pittsburgh/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/sacramento/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/sanfrancisco/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/index/general.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/index/image.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/news.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/index/video.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/index/collection.xml Sitemap: https://www.cbsnews.com/brandstudio/xml-sitemap/video-news.xml Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-US Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-NY Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-LA Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-BOS Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-SF Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-MIN Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-PIT Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-PHI Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-SAC Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-DEN Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-DAL Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-CHI Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-BAL Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-MIA Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-DET Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-Nature Sitemap: https://www.cbsnews.com/live/sitemap.xml?fusionEdition=CBSN-CAR Sitemap: https://www.cbsnews.com/xml-sitemap/cbs-essentials.xml User-agent: GPTBot Disallow: / User-agent: MAZBot Disallow: / User-agent: panscient.com Disallow: / User-agent: proximic Disallow: