Free tools

Robots.txt Examples

Robots.txt file content for chegg.com.

Robot.txt file for: chegg.com

      User-agent: ia_archiver
Disallow: /

User-agent: *
Disallow: /Search
Disallow: /_ajax
Disallow: /study/_ajax
Disallow: /auth
Disallow: /login
Disallow: /login/
Disallow: /account
Disallow: /checkout
Disallow: /homework-help/landing/7-day
Disallow: /homework-help/landing/author
Disallow: /homework-help/landing/general
Disallow: /homework-help/landing/hwh
Disallow: /homework-help/landing/offer
Disallow: /homework-help/search?
Disallow: /my/
Disallow: /myaccount
Disallow: /psearch
Disallow: /render
Disallow: /returnlabel
Disallow: /search
Disallow: /selltextbooks/search
Disallow: /sell-textbooks/search
Disallow: /s/
Disallow: /shoppingcart
Disallow: /signup
Disallow: /extendables
Disallow: /tracking
Disallow: /tutors/signin/
Disallow: /forgotpassword
Disallow: /test-prep/kaplan/redirect
Disallow: /flashcards/search/
Disallow: /salliemae
Disallow: /salliemae/termsandconditions
Disallow: /tutors/s/
Disallow: /tutors/ide/
Disallow: /campaign
Disallow: /textbooks/_ajax/log
Disallow: /tutors/api/v1/event/
Disallow: /schools/_ajax/log
Disallow: /tutors/publish/static/ssl-storage.html
Disallow: /flashcards/r/
Disallow: /*/page_view_event/
Disallow: /*/session_trace/
Disallow: /*/page_view_timing/
Disallow: /*/metrics/
Disallow: /*/ajax/
Disallow: /*/jserrors/
Disallow: /*/aggregate/
Disallow: /tools/writing-center
Disallow: /course-materials/
Disallow: /chat
Disallow: /learn
Disallow: /homework-help/definitions
Disallow: /flashcards/
Disallow: /*/293KxffvUkJMLzB21vpMzE.json
Disallow: /pla/*
Disallow: /prepsheet-feedback/*

User-agent: Mediapartners-Google
Allow: /