Free tools
Robots.txt Examples
Robots.txt file content for aafp.org.
Robot.txt file for: aafp.org
# By accessing the AAFP website you agree to AAFP's terms of use, available at # https://www.aafp.org/about-site/requests/terms.html # # No robots were harmed in the creation of this website. User-agent: * Disallow: /chapter Disallow: /login Disallow: *aafpvlogin* Disallow: /afp/accessories* Disallow: /fpm/accessories* Disallow: /FPMprinter* Disallow: /fpm/FPMprinter* Disallow: /afp/AFPprinter* Disallow: /afp/viewRelatedDocumentsByMesh.htm* Disallow: /fpm/viewRelatedDocumentsByMesh.htm* Disallow: *.html?null$ Disallow: /fpr/* Disallow: /PreBuilt/* Disallow: /online/etc/medialib* Disallow: /cme/subscriptions/fp-essentials/editions.html?contentpar* Disallow: /dam/AAFP/documents/cme* Disallow: /dam/Documentation* Disallow: /afp/1998/0301/p908.html Disallow: *contentpar_gridblock* Disallow: *content.pdflist.html$ Disallow: /usertesting* Disallow: *?plainText Disallow: /dam/*.pdf$ Disallow: /dam/*.xls$ Disallow: /dam/*.xlsx$ Disallow: /dam/*.docx$ Disallow: /dam/*.doc$ Disallow: /dam/*.ppt$ Disallow: /dam/*.pptx$ User-agent: OmtrBot/1.0 Disallow: Sitemap: https://www.aafp.org/sitemap.xml