Free tools
Robots.txt Examples
Robots.txt file content for nationalgeographic.com.
Robot.txt file for: nationalgeographic.com
User-agent: * # Directory Excludes Disallow: /search* Disallow: /tv/watch-live/ Disallow: /tv/browse Disallow: /tv/movies-and-specials/ Disallow: /tv/shows/ Disallow: /tv/my-profile Disallow: /api/federation/ # Legacy Excludes Disallow: /cgi-bin/ Disallow: /*.swf$ Disallow: /*eid= Disallow: /*email= Disallow: /*intcmp= Disallow: /*ngc= Disallow: /*referrer= Disallow: /*widgets= Disallow: /admin* Disallow: /ads/ Disallow: /au/ Disallow: /cp/ Disallow: /ebooklets/ Disallow: /event.ng/ Disallow: /image/ Disallow: /in/ Disallow: /magazines/l/multisubs/* Disallow: /magazines/all-banners-2* Disallow: /magazines/all-banners* Disallow: /magazines/all-dl* Disallow: /magazines/all-flyout* Disallow: /magazines/all-Ip* Disallow: /magazines/all-lp* Disallow: /magazines/all-sem* Disallow: /magazines/all-subs* Disallow: /magazines/Ip* Disallow: /magazines/lp* Disallow: /magazines/natgeomagazines-all* Disallow: /magazines/natgeomagazines* Disallow: /magazines/print-sem* Disallow: /media/interactive/cdrom/onehundredandeightyears/ Disallow: /media/tv/channel/credits/ Disallow: /pc/ Disallow: /resources/ngo/maps/atlas/ Disallow: /topics/ # Disallow Directives for AI Web Crawlers User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Claude-Web Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GoogleOther Disallow: / User-agent: GoogleOther-Image Disallow: / User-agent: GoogleOther-Video Disallow: / User-agent: GPTBot Disallow: / User-Agent: PerplexityBot Disallow: / # Announce Sitemap Locations Sitemap: https://www.nationalgeographic.com/sitemaps/sitemap.xml Sitemap: https://nationalgeographic.com/tv/sitemapindex-episodes.xml Sitemap: https://nationalgeographic.com/tv/sitemapindex-showmap.xml Sitemap: https://nationalgeographic.com/tv/sitemapindex-videomap.xml Sitemap: https://nationalgeographic.com/expeditions/sitemap.xml