Free tools
Robots.txt Examples
Robots.txt file content for bloomberg.com.
Robot.txt file for: bloomberg.com
# Bot rules: # 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm. # 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law. # 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law. # If you can read this then you should apply here https://www.bloomberg.com/careers/ User-agent: * Disallow: /polska Allow: /account/newsletters Disallow: /account/* Disallow: /tosv*.html Disallow: /search Disallow: /company/search/ Disallow: /professional/search/ Disallow: /impact/search/ Disallow: /ux/search/ Disallow: /wnwi/search/ Disallow: /gei/search/ Disallow: /impact/search/ Disallow: /netzeropathfinders/search/ Disallow: /notices/search/ Disallow: /distribution/search/ Disallow: /ukinnovators/search/ Disallow: /latam/search/ Disallow: /faq/search/ Disallow: /tc/search/ Disallow: /subscriptions/group/manage/ Disallow: /preview/lineup Disallow: /preview/articles Disallow: /explore/ Disallow: /press-releases/ Disallow: /artemis/ Disallow: /sessions-publisher/ User-agent: Google-Extended Disallow: / User-agent: Mediapartners-Google Disallow: /about/careers Disallow: /about/careers/ Disallow: /offlinemessage/ Disallow: /apps/fbk Disallow: /bb/newsarchive/ Disallow: /apps/news User-agent: Spinn3r Disallow: /podcasts/ Disallow: /feed/podcast/ Disallow: /bb/avfile/ User-agent: Googlebot-News Disallow: /sponsor/ Disallow: /news/sponsors/* Disallow: /news/terminal/* User-agent: Twitterbot Allow: /en/news/thp User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://www.bloomberg.com/sitemap.xml Sitemap: https://www.bloomberg.com/feeds/sitemap_news.xml Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/bpol/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/businessweek/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/citylab/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/green/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/technology/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_securities_index.xml Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_profiles_company_index.xml Sitemap: https://www.bloomberg.com/billionaires/sitemap.xml Sitemap: https://www.bloomberg.com/feeds/bbiz/sitemap_news.xml Sitemap: https://www.bloomberg.com/feeds/dynamic/private-company-index.xml Sitemap: https://www.bloomberg.com/feeds/dynamic/person-index.xml Sitemap: https://www.bloomberg.com/feeds/curated/feeds/graphics_news.xml Sitemap: https://www.bloomberg.com/feeds/curated/feeds/graphics_sitemap.xml Sitemap: https://www.bloomberg.com/feeds/equality/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/wealth/sitemap_index.xml Sitemap: https://www.bloomberg.com/multimedia/sitemap.xml Sitemap: https://www.bloomberg.com/feeds/curated/bbiz_sitemap.xml Sitemap: https://www.bloomberg.com/feeds/pursuits/sitemap_index.xml Sitemap: https://www.bloomberg.com/feeds/quicktake/sitemap_index.xml