Free tools
Robots.txt Examples
Robots.txt file content for gamespot.com.
Robot.txt file for: gamespot.com
# robots.txt for https://www.gamespot.com/ User-agent: AhrefsBot Disallow: / User-agent: AwarioRssBot User-agent: AwarioSmartBot Disallow: / User-agent: barkrowler Disallow: / User-agent: BLEXBot Disallow: / User-agent: CCBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: dotbot Disallow: / User-agent: GPTBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: * Disallow: */?comment_page=* Allow: /api/$ Disallow: /api/ Disallow: /chat/ Disallow: /community/ Disallow: /email/ Disallow: /feeds/* Disallow: /forgot-password/ Disallow: /g00/* Disallow: /genre/ Disallow: /images/ Disallow: /jsonsearch/ Disallow: /latest-amp-article/ Disallow: /login/ Disallow: /login-signup/ Disallow: /notifications/ Disallow: /pages/ Disallow: /platform/ Disallow: /postRender Disallow: /profile/*/blog/ Disallow: /profile/*/reviews/ Disallow: /profile/*/trophies/ Disallow: /search/ Disallow: /signup/ Disallow: /tags/ Disallow: /test/ Disallow: /theme/ Disallow: /trax/ Disallow: /users/