Free tools
Robots.txt Examples
Robots.txt file content for harvard.edu.
Robot.txt file for: harvard.edu
# START YOAST BLOCK # --------------------------- User-agent: * Disallow: Sitemap: https://www.harvard.edu/sitemap_index.xml Sitemap: https://www.harvard.edu/president/sitemap_index.xml Sitemap: https://www.harvard.edu/media-relations/sitemap_index.xml Sitemap: https://www.harvard.edu/admissionscase/sitemap_index.xml Sitemap: https://www.harvard.edu/climate-and-sustainability/sitemap_index.xml Sitemap: https://www.harvard.edu/wellbeing/sitemap_index.xml Sitemap: https://www.harvard.edu/guidelines/sitemap_index.xml Sitemap: https://www.harvard.edu/vice-provost-for-special-projects/sitemap_index.xml Sitemap: https://www.harvard.edu/ai/sitemap_index.xml # --------------------------- # END YOAST BLOCK