Free tools

Robots.txt Examples

Robots.txt file content for dictionary.com.

Robot.txt file for: dictionary.com

      # Production robots.txt

User-agent: GPTBot
Disallow: /

User-agent: deepcrawl
Disallow: /

User-agent: seerbot/1.0
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /5480.iac.
Disallow: /go/
Disallow: /audio.html/
Disallow: /houseads/
Disallow: /askhome/
Disallow: /23219321/iac.
Disallow: /*,*
Disallow: /browse-2/

Allow: /
Sitemap: https://www.dictionary.com/sitemap-dict-index.xml
Sitemap: https://www.dictionary.com/e/sitemap_index.xml