Free tools

Robots.txt Examples

Robots.txt file content for allocine.fr.

Robot.txt file for: allocine.fr

      User-Agent:*
Disallow: /rechercher/
Disallow: /salle/recherche/
Disallow: /blogvision/
Disallow: /communaute/membre/reportabuseforum_gen_nofil=*
Disallow: /*?*iframe*
Disallow: /ws/AcVisiondata.ashx?*
Disallow: /ws/AcVisiondataV4.ashx?*
Disallow: /ws/ProfilLightGetModerationLink.ashx?*
Disallow: /ws/geocoding.ashx
Disallow: /ws_v7/geocoding.ashx
Disallow: /ws/WinnerListHandler.ashx
Disallow: /ws/PostRollTrailer.ashx?*
Disallow: /ws/partnerproxy.ashx?*
Disallow: /ws_v7/Disqus.asmx/
Disallow: /ws_v7/get_complete_photos_list.ashx?*
Disallow: /120157152/
Disallow: /doubleclick/
Disallow: /_/
Disallow: /_video/partner/

User-Agent: seekbot
Disallow: /

User-Agent: Exabot
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: Cliqzbot
Disallow: /