Free tools
Robots.txt Examples
Robots.txt file content for redalyc.org.
Robot.txt file for: redalyc.org
User-agent: * Disallow: /redalyc/indexfiles Disallow: /redalyc/lib Disallow: /redalyc/xmlindex Disallow: /indicador/ Allow: /redalyc/media # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* User-agent: AhrefsBot Crawl-delay: 50 Disallow: / User-agent: msnbot Crawl-delay: 10 Disallow: / User-agent: dotbot Disallow: / User-agent: turnitinbot Disallow: / User-agent: Bingbot Crawl-delay: 1 Disallow: User-agent: IstellaBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: Yandex Allow: / User-agent: CCBot Disallow: / Sitemap: https://www.redalyc.org/sitemaps/sitemap-index.xml