Free tools

Robots.txt Examples

Robots.txt file content for scielo.br.

Robot.txt file for: scielo.br

      # Block admin to all crawlers
User-agent: *
Disallow: /admin

# Allow all crawlers
User-agent: *
Allow: /