Free tools

Robots.txt Examples

Robots.txt file content for free.fr.

Robot.txt file for: free.fr

      # Instructions sur les badbots
# Alexa Archive, IBM Watson, Facebook, Search metrics
# Linux, Base de donnÃ©e Ã©tudiante, Aspirateur web
User-agent: ia_archiver
User-agent: ScoutJet
User-agent: Facebot
User-agent: SearchmetricsBot
User-Agent: TurnitinBot
User-Agent: HTTrack
Disallow: /

# Instructions pour les autres bots
User-agent: *
Allow: /
Disallow: /apps/
Disallow:

Sitemap: https://www.free.fr/sitemap.xml