Free tools
Robots.txt Examples
Robots.txt file content for rochester.edu.
Robot.txt file for: rochester.edu
User-agent: atraxbot/0.2 Disallow: / User-agent: * Disallow: /adv/advancement/ Disallow: /report/ Disallow: /ITS/reports/ Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /ITS/web/group/dev/ Disallow: /calendar/ Disallow: /calendar/upload.php Disallow: /calendar/epicker.php Disallow: /beta/ Disallow: /ur-cgi-bin/ Disallow: /its/dcs/forms/ Disallow: /ITS/dcs/forms/ Disallow: /college/CCAS/clusters/cluster_search7.html Disallow: /college/ccas/clusters/cluster_search7.html Disallow: /its/acs/oge_verifies_form.html Disallow: /college/alumni/reunion2011/classes/ Disallow: /giving/gec/roll Disallow: /giving/gec/roll2 Disallow: /college/alumni/reunion/classes/ Disallow: /2013/donors Disallow: /@rochester/archive/issue.php?iss=2701 Disallow: /currents/V41/N02/Sloan.html Disallow: /college/ugresearch/ur-upp/ Disallow: /poc-tech/ Disallow: /working/hr/jobs/feeds/ Disallow: /wp-admin/ Disallow: /policies/all/ User-agent: ReadableBot Disallow: