Free tools

Robots.txt Examples

Robots.txt file content for thewrap.com.

Robot.txt file for: thewrap.com

      User-agent: Yandexbot
Disallow: /
User-agent: MojeekBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: Linespider
Disallow: /
User-agent: *
Crawl-delay: 30
Disallow:
Disallow: /tag/
Allow: /wp-admin/admin-ajax.php
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /search/
Disallow: /wp-login.php
Disallow: /activate/
Disallow: /?s=
Sitemap: https://www.thewrap.com/sitemap.xml