Free tools

Robots.txt Examples

Robots.txt file content for apa.org.

Robot.txt file for: apa.org

              User-agent: *
        Disallow: /secure/
        Disallow: /*/secure/
        Disallow: /*.bin$
        Disallow: /related
        Disallow: /search
        Disallow: /*/search.html
        Disallow: /*/browse.html
        Disallow: /*/search?
        Disallow: /*/browse?
        Disallow: /Content
        Allow: /Content/Images
        User-agent: sogou spider
        Disallow: /
        User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
        Disallow: /
        User-agent: Sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
        Disallow: /
        User-agent: Sougou+web+spider
        Disallow: /
        Sitemap: https://www.apa.org/sitemap.xml