Free tools
Robots.txt Examples
Robots.txt file content for canada.ca.
Robot.txt file for: canada.ca
#Government of Canada / Gouvernement du Canada #Block AEM folders for CRA User-agent: * Disallow: /content/dam/cra-arc/formspubs/ Disallow: /en/revenue-agency/web-services-test/ Disallow: /fr/agence-revenu/test-web-services/ #Search pages do not need to be crawled Disallow: /en/sr/srb.html Disallow: /fr/sr/srb.html Disallow: /en/sr/srb/sra.html Disallow: /fr/sr/srb/sra.html Disallow: /en/*/search.html Disallow: /en/*/search/advanced-search.html Disallow: /fr/*/rechercher.html Disallow: /fr/*/rechercher/recherche-avancee.html Disallow: /en/*/menu/header.html Disallow: /fr/*/menu/header.html Disallow: /en/*/menu/footer.html Disallow: /fr/*/menu/footer.html Disallow: /en/*/menu.html Disallow: /fr/*/menu.html Disallow: /en/*/footer/contactinformation.html Disallow: /fr/*/footer/Coordonnees.html Disallow: /*/_jcr_content/par* Disallow: /en/service-canada/ Disallow: /fr/service-canada/ Disallow: /en/immigration-refugees-citizenship/services/reference-include/ Disallow: /fr/immigration-refugies-citoyennete/services/reference-inclusion/ #IRCC PDFs Disallow: /content/dam/ircc/documents/pdf/english/kits/guides/guide-0142-airlifted-afghanistan-pathway-pr.pdf Disallow: /content/dam/ircc/documents/pdf/francais/trousses/guides/guide-0142-avion-afghanistan-voie-acces-rp.pdf Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm0143e.pdf Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm0143f.pdf Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm0144e.pdf Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm0144f.pdf Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm5444/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm5444/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm5644/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm5644/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm5475/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm5475/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/imm5476/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/imm5476/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/irm0002/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/irm0002/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/irm0004/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/irm0004/ Disallow: /content/dam/ircc/documents/pdf/english/kits/forms/irm0005/ Disallow: /content/dam/ircc/documents/pdf/francais/trousses/form/irm0005/