Free tools

Robots.txt Examples

Robots.txt file content for hgtv.com.

Robot.txt file for: hgtv.com

      User-agent: *
Allow: /content/dam/
Allow: /content/hgtv-com/jcr:content/analytics.sitecatalyst.js
Disallow: /content/
Allow: /libs/granite/
Disallow: /libs/
Disallow: /services/
Disallow: /search/
Disallow: /features/apple-news/
Disallow: /shows/tv-schedule.*
Disallow: *.embed
Allow: *.lazy-fetch*
Allow: *hotspot*

User-agent: Mediapartners-Google
Disallow:

User-agent: Pinterest
Crawl-delay: 5

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

Sitemap: https://www.hgtv.com/sitemaps/sitemap_hgtv_index.xml