Free tools

Robots.txt Examples

Robots.txt file content for instructables.com.

Robot.txt file for: instructables.com

      Sitemap: https://www.instructables.com/sitemap.xml

User-agent: Mediapartners-Google*
Disallow:

User-agent: Slurp
Crawl-delay: 1

User-agent: AdsBot-Google*
Disallow:

User-agent: *
Disallow: /*.pdf$
Disallow: /*.txt$
Disallow: /*.html$
Disallow: /file/*
Disallow: /howto/*
Disallow: /image/*