Free tools
Robots.txt Examples
Robots.txt file content for apa.org.
Robot.txt file for: apa.org
User-agent: * Disallow: /secure/ Disallow: /*/secure/ Disallow: /*.bin$ Disallow: /related Disallow: /search Disallow: /*/search.html Disallow: /*/browse.html Disallow: /*/search? Disallow: /*/browse? Disallow: /Content Allow: /Content/Images User-agent: sogou spider Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sougou+web+spider Disallow: / Sitemap: https://www.apa.org/sitemap.xml