Free tools
Robots.txt Examples
Robots.txt file content for teacherspayteachers.com.
Robot.txt file for: teacherspayteachers.com
User-agent: * Sitemap: https://www.teacherspayteachers.com/Sitemap_index.xml Disallow: /*.pdf$ Disallow: /*.xls$ Disallow: /*.ppt$ Disallow: /*.doc$ Disallow: /*.rtf$ Disallow: /*.swf$ Disallow: /errors/ Disallow: /store/*?* Disallow: /Store/*/Search Disallow: /Store/*/Search* Disallow: /Store/*/Order* Disallow: /Download/* Disallow: /carts/ajax_index Disallow: /Cart/Add* Disallow: /Login/* Disallow: /Login?* Disallow: /browse?order= Disallow: /FreeDownload/* Disallow: /graph/graphql* Disallow: /graph/graphiql.html* Disallow: /users Disallow: /users/* Disallow: /Cart Disallow: /Cart/* Disallow: /Wish-List Disallow: /Wish-List/* Disallow: /University/ Disallow: /University/* Disallow: /university/ Disallow: /university/* Disallow: /widgets/show/* Disallow: /widgets/get/* User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: CCBot Disallow: / User-agent: Yandex Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Slurp Crawl-delay: 20 User-agent: Yahoo! Slurp Crawl-delay: 20 User-agent: Twitterbot User-agent: Applebot-Extended Disallow: /