Free tools

Robots.txt Examples

Robots.txt file content for gymglish.com.

Robot.txt file for: gymglish.com

      # robots.txt for http://www.gymglish.com/
# More on http://www.robotstxt.org
sitemap: https://www.gymglish.com/sitemapindex.xml
User-agent: *

# Exclude some URLs
Disallow: /wb/www/notretemps-www/
Disallow: /nosquid/
Disallow: /www/
Disallow: /virtual/
#Disallow: /partner/ -> handled using meta tags
Disallow: */cpartner/
Disallow: */partner/
Disallow: */lps/
Disallow: /api/

# Allow online shop, but disallow rest of workbook
Allow: /workbook/shop
Disallow: /workbook/
Disallow: */workbook/
Disallow: */showlesson
Disallow: /backoffice/

# Disallow media
Disallow: /audios/
Disallow: /videos/
Disallow: /opensource/

# Referral links & registration pages
Disallow: /r/

# Exclude some documents
Disallow: /documents/newsletter/
Disallow: /documents/CP/
Disallow: /documents/gymglish_examples/