Free tools

Robots.txt Examples

Robots.txt file content for timesofindia.com.

Robot.txt file for: timesofindia.com

      User-agent: *
Allow: /
Disallow: /PDATOI/
Disallow: /topic/quote/
Disallow: /*cms.dll*
Disallow: /*rssarticleshow*
Disallow: /topic/photo/
Disallow: /default.cms
Disallow: /default1.cms
Disallow: /artshowcmt/
Disallow: */opinions/
Disallow: /breakingnews/toihome.html
Disallow: /topic/article/
Disallow: /logtopickeywords*
Disallow: /new_cmtofart*
Disallow: /pmcomment/
Disallow: /feeds/
Disallow: */configspace/*
Disallow: /configspace/maintenance.html
Disallow: /topiclist/
Disallow: */pmredirectshow/*
Disallow: */stoi/*
Disallow: */moviearticlelistalphabet1/*
Disallow: */cinema-entertainment/*
Disallow: */life-style-landing/*
Disallow: *navlvl=
Disallow: /lspoll/
Disallow: *query=
Disallow: /beta_hotlydbt/
Disallow: *defalpha=
Disallow: *,page
Disallow: /new_cmtofart2_nit_v1/
Disallow: *prtpage
Disallow: */examshow/*
Disallow: */speednewsshow/*
Disallow: */speednewsbytopic/*
Disallow: */itslideshow/*
Disallow: */vsarticleshow/*
Disallow: */articleshow_amitr/*
Disallow: */pmtoinavigation/*
Disallow: */mvopinions/*
Disallow: */opinionslogged/*
Disallow: */rodate.js
Disallow: */load.js
Disallow: */cookie.js
Disallow: */jquery.js
Disallow: */navigation.js
Disallow: */pgtrack.js
Disallow: */lazy.js
Disallow: */breakingnews.js
Disallow: */rodate.js
Disallow: */mail/*
Disallow: */profileshow/* 
Disallow: */tpoint_articleshow/*
Disallow: */trivia/*
Disallow: */articleshowajax/*
Disallow: /articlecomment/
Disallow: /artshowcmt/
Disallow: /7176/TOI_Travel/
Disallow: /search
Disallow: /fbpostaction.cms?article=
Disallow: /7176/TOI_Travel_Test/TOI_Travel_TOP_TST
Disallow: /postmail.cms
Disallow: /mostviewed
Disallow: /related
Disallow: /comments
Disallow: *Jai-Mrug*
Disallow: */affiliate_amazon.cms*
Disallow: *-mostviwed*
Disallow: */techgadgets/*
Disallow: */covid-19-delhi/*
Disallow: */covid-19-bangalore/*
Disallow: */covid-19-hyderabad/*
Disallow: */covid-19-kolkata/*
Disallow: */covid-19-mumbai/*
Disallow: */covid-19-across-world/*
Disallow: */crshow/*
Disallow: */briefs/articleid-*
Disallow: /intarnel/*
Disallow: */dynalist/*
Disallow: */amp_msp_amazon_gadget.cms*
Disallow: */affiliates_content_iframe.cms*
Disallow: */affiliates_content.cms*
Disallow: */affiliates_content_iframe*
Disallow: */affiliates_library*
Disallow: */affiliates_content*
Disallow: */ifrmwdt/*
Disallow: *articleshowbriefs*
Disallow: *quickstirshow*
Disallow: *mobile=no*
Disallow: *etarticleshow*
Disallow: *?mobile=no*
Disallow: *homeajax_nav.cms?*
Disallow: *articleshowbriefs?*
Disallow: *affiliates_content_iframe_v2*
Disallow: *?frmapp=yes*
Disallow: *embedvideo.cms*
Disallow: *videodash.cms*
Disallow: *etopneinappamp.cms*
Disallow: *as_comments*
Disallow: *RBP_HOME_URL*
Disallow: *?comments*
Disallow: *?showcomment*
Disallow: *?cmfid*
Disallow: *?pg=*
Disallow: *?ps=*
Disallow: *articleshow_new*
Disallow: *articleshowprint*
Disallow: /microsite_v2
Disallow: */undefined$
Disallow: */articleshowhsbc/
Disallow: */storylist/
Disallow: *asrelated.cms*
Disallow: *amp_next_page_ga.cms*
Disallow: */travel/search?*

User-agent: Googlebot-News
Allow: /
Disallow: /PDATOI/
Disallow: /topic/quote/
Disallow: /*cms.dll*
Disallow: /*rssarticleshow*
Disallow: /topic/photo/
Disallow: /default.cms
Disallow: /default1.cms
Disallow: /artshowcmt/
Disallow: */opinions/
Disallow: /breakingnews/toihome.html
Disallow: /topic/article/
Disallow: /logtopickeywords*
Disallow: /new_cmtofart*
Disallow: /pmcomment/
Disallow: /feeds/
Disallow: */configspace/*
Disallow: /configspace/maintenance.html
Disallow: /topiclist/
Disallow: */pmredirectshow/*
Disallow: */stoi/*
Disallow: */moviearticlelistalphabet1/*
Disallow: */cinema-entertainment/*
Disallow: */life-style-landing/*
Disallow: *navlvl=
Disallow: /lspoll/
Disallow: *query=
Disallow: /beta_hotlydbt/
Disallow: *defalpha=
Disallow: *,page
Disallow: /new_cmtofart2_nit_v1/
Disallow: *prtpage
Disallow: */examshow/*
Disallow: */speednewsshow/*
Disallow: */speednewsbytopic/*
Disallow: */itslideshow/*
Disallow: */vsarticleshow/*
Disallow: */articleshow_amitr/*
Disallow: */pmtoinavigation/*
Disallow: */mvopinions/*
Disallow: */opinionslogged/*
Disallow: */rodate.js
Disallow: */load.js
Disallow: */cookie.js
Disallow: */jquery.js
Disallow: */navigation.js
Disallow: */pgtrack.js
Disallow: */lazy.js
Disallow: */breakingnews.js
Disallow: */rodate.js
Disallow: */mail/*
Disallow: */profileshow/* 
Disallow: */tpoint_articleshow/*
Disallow: */trivia/*
Disallow: */articleshowajax/*
Disallow: /articlecomment/
Disallow: /artshowcmt/
Disallow: /7176/TOI_Travel/
Disallow: /search
Disallow: /fbpostaction.cms?article=
Disallow: /7176/TOI_Travel_Test/TOI_Travel_TOP_TST
Disallow: /postmail.cms
Disallow: /mostviewed
Disallow: /related
Disallow: /comments
Disallow: *Jai-Mrug*
Disallow: */affiliate_amazon.cms*
Disallow: *-mostviwed*
Disallow: */techgadgets/*
Disallow: */covid-19-delhi/*
Disallow: */covid-19-bangalore/*
Disallow: */covid-19-hyderabad/*
Disallow: */covid-19-kolkata/*
Disallow: */covid-19-mumbai/*
Disallow: */covid-19-across-world/*
Disallow: */crshow/*
Disallow: */briefs/articleid-*
Disallow: /intarnel/*
Disallow: */dynalist/*
Disallow: */amp_msp_amazon_gadget.cms*
Disallow: */affiliates_content_iframe.cms*
Disallow: */affiliates_content.cms*
Disallow: */affiliates_content_iframe*
Disallow: */affiliates_library*
Disallow: */affiliates_content*
Disallow: */ifrmwdt/*
Disallow: *articleshowbriefs*
Disallow: *quickstirshow*
Disallow: *mobile=no*
Disallow: *etarticleshow*
Disallow: *?mobile=no*
Disallow: *homeajax_nav.cms?*
Disallow: *articleshowbriefs?*
Disallow: *affiliates_content_iframe_v2*
Disallow: *?frmapp=yes*
Disallow: *embedvideo.cms*
Disallow: *videodash.cms*
Disallow: *etopneinappamp.cms*
Disallow: *as_comments*
Disallow: *RBP_HOME_URL*
Disallow: *?comments*
Disallow: *?showcomment*
Disallow: *?cmfid*
Disallow: *?pg=*
Disallow: *?ps=*
Disallow: *articleshow_new*
Disallow: *articleshowprint*
Disallow: /microsite_v2
Disallow: */undefined$
Disallow: */articleshowhsbc/
Disallow: */storylist/
Disallow: *asrelated.cms*
Disallow: *amp_next_page_ga.cms*
Disallow: */travel/search?*

User-agent: GPTBot
Disallow: /