Free tools
Robots.txt Examples
Robots.txt file content for aajtak.in.
Robot.txt file for: aajtak.in
User-Agent: * Allow: / Disallow: /get_graph_json.php Disallow: /live-score/seriesLiveList/ Disallow: /live-score/seriesRecentList/ Disallow: /live-score/seriesUpcomingList/ Disallow: /live-score/venueTeamListBySeriesId/* Disallow: /live-score/series_team_venue_filter/ Disallow: /live-score/recent/home/widget/?theme=1&header=no Disallow: /live-score/upcoming/home/ Disallow: /live-score/cricket/teams_profileApiData/* Disallow: /live-score/cricket/head_to_headApiData/* Disallow: /live-score/cricket/full_scorecardApiData/* Disallow: /live-score/cricket/comm/ Disallow: /live-score/cricket/commentary/ Disallow: /live-score/fixtures/live_score_flipApiData/* Disallow: /live-score/fixtures/live_scoreApiData/* Disallow: /live-score/fixtures/widget/fixtures_swapper/* Disallow: /live-score/cricket/auction_ajax/* Disallow: /search/* Disallow: /suniye-neta-ji/video/* Disallow: /superfast/* Disallow: /shopping/* User-agent: Googlebot-News Disallow: /impact-feature/* Disallow: /visualstories/preview.php Sitemap: https://www.aajtak.in/rssfeeds/sitemap.xml Sitemap: https://www.aajtak.in/rssfeeds/news-sitemap.xml Sitemap: https://www.aajtak.in/rssfeeds/video-sitemap.xml Sitemap: https://www.aajtak.in/rssfeeds/image-sitemap.xml Sitemap: https://www.aajtak.in/visualstories/sitemap.xml User-agent: CCBot Disallow: /