Free tools

Robots.txt Examples

Robots.txt file content for worldbank.org.

Robot.txt file for: worldbank.org

      User-agent: *
Allow:*
Disallow: */apps/*
Disallow: */wbg/services/*
Disallow: */en/webarchives/archive*
Disallow: */misc*
Disallow: */conf/*
Disallow: */downloadstats/*
Disallow: */wbg/aem/*
Disallow: */en/search*
Disallow: */content/search*
Disallow: */en/programs/umbrellafacilityforgenderequality/brief/ufge-partnership-council-meeting
Sitemap: https://www.worldbank.org/sitemap.xml