Free tools

Robots.txt Examples

Robots.txt file content for teacherspayteachers.com.

Robot.txt file for: teacherspayteachers.com

      User-agent: *
Sitemap: https://www.teacherspayteachers.com/Sitemap_index.xml

Disallow: /*.pdf$
Disallow: /*.xls$
Disallow: /*.ppt$
Disallow: /*.doc$
Disallow: /*.rtf$
Disallow: /*.swf$
Disallow: /errors/
Disallow: /store/*?*
Disallow: /Store/*/Search
Disallow: /Store/*/Search*
Disallow: /Store/*/Order*
Disallow: /Download/*
Disallow: /carts/ajax_index
Disallow: /Cart/Add*
Disallow: /Login/*
Disallow: /Login?*
Disallow: /browse?order=

Disallow: /graph/graphql*
Disallow: /graph/graphiql.html*

Disallow: /users
Disallow: /users/*
Disallow: /Cart
Disallow: /Cart/*
Disallow: /Wish-List
Disallow: /Wish-List/*

Disallow: /University/
Disallow: /University/*
Disallow: /university/
Disallow: /university/*

Disallow: /widgets/show/*
Disallow: /widgets/get/*

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Slurp
Crawl-delay: 20

User-agent: Yahoo! Slurp
Crawl-delay: 20

User-agent: Twitterbot