Robots.txt for bab.la
textile
User-agent: *
Disallow: /auth/xshare
Disallow: /*/spellcheck/
User-agent: Bingbot
User-agent: ias_crawler
User-agent: ias_wombles
User-agent: SirdataBot
Crawl-delay: 1
User-agent: Verity/1.1
Crawl-delay: 3
User-agent: Keybot Translation-Search-Machine
User-agent: Covario-IDS
User-agent: 008
Disallow:/