Free tools
Robots.txt Examples
Robots.txt file content for worldbank.org.
Robot.txt file for: worldbank.org
User-agent: * Allow:* Disallow: */apps/* Disallow: */wbg/services/* Disallow: */en/webarchives/archive* Disallow: */misc* Disallow: */conf/* Disallow: */downloadstats/* Disallow: */wbg/aem/* Disallow: */en/search* Disallow: */content/search* Disallow: */en/programs/umbrellafacilityforgenderequality/brief/ufge-partnership-council-meeting Sitemap: https://www.worldbank.org/sitemap.xml