Robots.txt for thewrap.com
User-agent: Yandexbot Disallow: / User-agent: MojeekBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Linespider Disallow: / User-agent: * Crawl-delay: 30 Disallow: Allow: /wp-admin/admin-ajax.php Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-login.php Sitemap: https://www.thewrap.com/sitemap.xml