Free tools

Robots.txt Examples

Robots.txt file content for gamespot.com.

Robot.txt file for: gamespot.com

      # robots.txt for https://www.gamespot.com/
User-agent: AhrefsBot
Disallow: /

User-agent: AwarioRssBot
User-agent: AwarioSmartBot
Disallow: /

User-agent: barkrowler
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: *
Disallow: */?comment_page=*
Allow: /api/$
Disallow: /api/
Disallow: /chat/
Disallow: /community/
Disallow: /email/
Disallow: /feeds/*
Disallow: /forgot-password/
Disallow: /g00/*
Disallow: /genre/
Disallow: /images/
Disallow: /jsonsearch/
Disallow: /latest-amp-article/
Disallow: /login/
Disallow: /login-signup/
Disallow: /notifications/
Disallow: /pages/
Disallow: /platform/
Disallow: /postRender
Disallow: /profile/*/blog/
Disallow: /profile/*/reviews/
Disallow: /profile/*/trophies/
Disallow: /search/
Disallow: /signup/
Disallow: /tags/
Disallow: /test/
Disallow: /theme/
Disallow: /trax/
Disallow: /users/