Free tools

Robots.txt Examples

Robots.txt file content for indiatimes.com.

Robot.txt file for: indiatimes.com

      #robots.txt 

User-agent: *
Allow: /
Disallow: *.php*
Disallow: */7176/*
Disallow: */whatshot/*
Disallow: */wattpad/*
Disallow: */uc_feed/*
Disallow: */brief/*
Disallow: */search/*
Disallow: */user/*
Disallow: */idtoin/*
Disallow: */video_player/*
Disallow: */blog/*
Disallow: */content/*
Disallow: */seoarchive/*
Disallow: */trends/audios/*
Disallow: */trends/videos/*
Disallow: *sw.js*
Disallow: *{*
Disallow: *&*
Disallow: *?utm_source*
Disallow: *?ref=*
Disallow: *?v=*
Disallow: *?google_editors_picks*
Disallow: *?no_fullscreen_*
Disallow: *?symbol=*
Disallow: */?date=*
Disallow: *?fb_comment_id=*
Disallow: *?_twitter*
Disallow: */2011/*
Disallow: */2012/*
Disallow: */2013/*
Disallow: */2014/*
Disallow: */2015/*
Disallow: */2016/*
Disallow: */2017/*
Disallow: */2018/*
Disallow: */2019/*
Disallow: */2020/*
Disallow: */2021/*
Disallow: */2022/*
Disallow: */2023/*
Disallow: */2024/*

#Sitemaps
Sitemap:https://www.indiatimes.com/sitemap.xml
Sitemap:https://www.indiatimes.com/sitemap-news.xml
Sitemap:https://www.indiatimes.com/sitemap-posts-index-2024.xml
Sitemap:https://www.indiatimes.com/hindi/sitemap-ampstories-2024.xml

User-agent: GPTBot
Disallow: /