Free tools

Robots.txt Examples

Robots.txt file content for aafp.org.

Robot.txt file for: aafp.org

      # By accessing the AAFP website you agree to AAFP's terms of use, available at
# https://www.aafp.org/about-site/requests/terms.html
#
# No robots were harmed in the creation of this website.

User-agent: *
Disallow: /chapter
Disallow: /login
Disallow: *aafpvlogin*
Disallow: /afp/accessories*
Disallow: /fpm/accessories*
Disallow: /FPMprinter*
Disallow: /fpm/FPMprinter*
Disallow: /afp/AFPprinter*
Disallow: /afp/viewRelatedDocumentsByMesh.htm*
Disallow: /fpm/viewRelatedDocumentsByMesh.htm*
Disallow: *.html?null$
Disallow: /fpr/*
Disallow: /PreBuilt/*
Disallow: /online/etc/medialib*
Disallow: /cme/subscriptions/fp-essentials/editions.html?contentpar*
Disallow: /dam/AAFP/documents/cme*
Disallow: /dam/Documentation*
Disallow: /afp/1998/0301/p908.html
Disallow: *contentpar_gridblock*
Disallow: *content.pdflist.html$
Disallow: /usertesting*
Disallow: *?plainText
Disallow: /dam/*.pdf$
Disallow: /dam/*.xls$
Disallow: /dam/*.xlsx$
Disallow: /dam/*.docx$
Disallow: /dam/*.doc$
Disallow: /dam/*.ppt$
Disallow: /dam/*.pptx$

User-agent: OmtrBot/1.0
Disallow:

Sitemap: https://www.aafp.org/sitemap.xml