Free tools

Robots.txt Examples

Robots.txt file content for ufl.edu.

Robot.txt file for: ufl.edu

      User-agent: *

Disallow: /wp-admin/
Disallow: /?page_id=5811&preview=true
Disallow: /xmlrpc.php
Disallow: /readme.html
Disallow: /refer/
Disallow: /cgi-bin/

Sitemap: https://www.ufl.edu/sitemap_index.xml