Free tools
Robots.txt Examples
Robots.txt file content for etymonline.com.
Robot.txt file for: etymonline.com
User-agent: Googlebot Disallow: /admin/ Disallow: /app/feedback User-agent: Bingbot Disallow: /classic Disallow: /classic/ Disallow: /admin/ Disallow: /app/feedback User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: SplitSignalBot Disallow: / User-agent: SemrushBot-COUB Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SemrushBot-CT Disallow: / User-agent: SemrushBot-BM Disallow: / User-agent: SemrushBot-SEOAB Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: dotbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: * Allow: /search Allow: /word/ Disallow: /web/login Disallow: /classic Disallow: /classic/ Disallow: /admin/ Disallow: /app/feedback Sitemap: https://www.etymonline.com/landings.xml Sitemap: https://www.etymonline.com/sitemap.xml Sitemap: https://www.etymonline.com/sitemap-article.xml Sitemap: https://www.etymonline.com/locale-sitemap/home.xml Sitemap: https://www.etymonline.com/sitemaps/words-locale.xml