Free tools
Robots.txt Examples
Robots.txt file content for cheatsheet.com.
Robot.txt file for: cheatsheet.com
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/certificates Disallow: /wp-includes/SimplePie Disallow: /wp-includes/Text Disallow: /wp-includes/*.php$ Disallow: /eyeblaster/addineyeV2.html Disallow: /wp-content/plugins Disallow: /Just-For-Test Disallow: /test/ Disallow: /recipes/ Disallow: /wine/ Disallow: /wine-jargon/ Disallow: /archive/ Disallow: /forum/ Disallow: /watchdog/ Disallow: /cronjob/ Disallow: /wp-content/uploads/wpsc/ Disallow: /downloads Disallow: /*Insert_Click_Track_URLhttp://ad.wsod.com/ Disallow: /*attachment/ Disallow: /*/*... Disallow: /1 Disallow: /pubclick Disallow: /newsletters/p/ Disallow: /members/p/ Disallow: /premium/p/ Disallow: /members-st/ Disallow: /?p=* Disallow: /mcs/* Disallow: /*/entertainment/page/*/ Allow: /ads.txt User-agent: Mediapartners-Google Allow: / User-agent: Googlebot-News Allow: / # Sitemap archive Sitemap: https://www.cheatsheet.com/news-sitemap.xml Sitemap: https://www.cheatsheet.com/sitemap.xml # START YOAST BLOCK # --------------------------- User-agent: * Disallow: # --------------------------- # END YOAST BLOCK