Free tools
Robots.txt Examples
Robots.txt file content for forbes.com.
Robot.txt file for: forbes.com
User-agent: * Allow: /ajax/follow_info* Allow: /ajax/getUniregUserData* Disallow: /following/ Disallow: /search/ Disallow: /follow/ Disallow: /ajax/ Disallow: /author/following/ Disallow: /blog/following/ Disallow: /json/ Disallow: /header/channels/ Disallow: /find-more/ Disallow: /preview/ Disallow: *wp-admin* Disallow: /typeahead/ Disallow: /pepe/ Disallow: /media-manager/ Disallow: /coupons/visit/* Disallow: /test/ Disallow: /advisor/*zx1fa* User-agent: googlebot-news Disallow: /sites/adobe/ Disallow: /sites/arizonacommerce/ Disallow: /sites/bmoharrisbank/ Disallow: /sites/businessaviation/ Disallow: /sites/capitalonespark/ Disallow: /sites/cdw/ Disallow: /sites/centurylink/ Disallow: /sites/dell/ Disallow: /sites/emc/ Disallow: /sites/ey/ Disallow: /sites/fedex/ Disallow: /sites/fidelity/ Disallow: /sites/gyro/ Disallow: /sites/hsbc/ Disallow: /sites/houlihanlawrence/ Disallow: /sites/hiscox/ Disallow: /sites/ibm/ Disallow: /sites/infosys/ Disallow: /sites/mdanderson/ Disallow: /sites/medidata/ Disallow: /sites/microsoft/ Disallow: /sites/microsoftdynamics/ Disallow: /sites/netapp/ Disallow: /sites/northwesternmutual/ Disallow: /sites/oppenheimerfunds/ Disallow: /sites/oracle/ Disallow: /sites/pace/ Disallow: /sites/raymondjames/ Disallow: /sites/sage/ Disallow: /sites/samsung/ Disallow: /sites/sap/ Disallow: /sites/sungardas/ Disallow: /sites/taboola/ Disallow: /sites/ups/ Disallow: /sites/delta/ Disallow: /sites/airbus/ Disallow: /sites/symantec/ Disallow: /sites/paypal/ Disallow: /sites/olddominion/ Disallow: /sites/salesforce/ Disallow: /sites/gapinternational/ Disallow: /sites/sprintbusiness/ Disallow: /sites/transamerica/ Disallow: /sites/constantcontact/ Disallow: /sites/robertocoin/ Disallow: /sites/canon/ Disallow: /sites/samsungbusiness/ Disallow: /sites/att/ Disallow: /sites/lasvegas/ Disallow: /sites/iowa/ Disallow: /sites/teradata/ Disallow: /sites/thehartford/ Disallow: /sites/statoil/ Disallow: /sites/cit/ Disallow: /sites/deloitte/ Disallow: /sites/huawei/ Disallow: /sites/relateiq/ Disallow: /sites/cftc/ Disallow: /sites/celebritycruises/ Disallow: /sites/universityofcalifornia/ Disallow: /sites/coxbusiness/ Disallow: /sites/pega/ Disallow: /sites/athenahealth/ Disallow: /sites/etrade/ Disallow: /sites/comcastbusiness/ Disallow: /sites/optum/ Disallow: /sites/seagate/ Disallow: /sites/rbcwealthmanagement/ Disallow: /sites/avigilon/ Disallow: /sites/citi/ Disallow: /sites/castlight/ Disallow: /sites/hyatt/ Disallow: /sites/johnhancock/ Disallow: /sites/koreanair/ Disallow: /sites/merrilllynch/ Disallow: /sites/ptc/ Disallow: /sites/toyota/ Disallow: /sites/unify/ Disallow: /sites/xerox/ Disallow: /sites/united/ Disallow: /sites/zenithwatches/ Disallow: /sites/jpmorganchase/ Disallow: /sites/ramcommercial/ Disallow: /sites/puremichigan/ Disallow: /sites/tmfgroup/ Disallow: /sites/usaa/ Disallow: /sites/emerson/ Disallow: /sites/aberdeen/ Disallow: /sites/kellogg/ Disallow: /sites/zurich/ Disallow: /sites/freeenterprise/ Disallow: /sites/allclearid/ Disallow: /sites/accenture/ Disallow: /sites/mercedesbenz/ Disallow: /sites/nice/ Disallow: /sites/riverbed/ Disallow: /sites/pimco/ Disallow: /sites/netjets/ Disallow: /sites/vmware/ Disallow: /sites/visa/ Disallow: /sites/statefarm/ Disallow: /sites/philips/ Disallow: /sites/adp/ Disallow: /sites/zenithwatches/ Disallow: /sites/citrix/ Disallow: /sites/gwcic/ Disallow: /sites/cargill/ Disallow: /sites/bankofsingapore/ Disallow: /sites/dbsbank/ Disallow: /sites/gradsoflife/ Disallow: /sites/internationaliotcouncil/ Disallow: /pega/customers-for-life Disallow: /celebritycruises/guide-to-the-globe Disallow: /sites/disneyinstitute/ Disallow: /sites/concertocloud/ Disallow: /sites/capitalonefutureedge/ Disallow: /sites/mahindracomviva/ Disallow: /sites/workday/ Disallow: /sites/colorado/ Disallow: /sites/charlesschwab/ Disallow: /sites/chicagobooth/ Disallow: /sites/morganstanley/ Disallow: /sites/opentext/ Disallow: /sites/icims/ Disallow: /sites/thriventmutualfunds/ Disallow: /sites/cbre/ Disallow: /sites/universityofphoenix/#57934e047260 Disallow: /sites/ppg/ Disallow: /sites/americanfunds/ Disallow: /sites/lilly/ Disallow: /sites/norton/ Disallow: /sites/qualityassurance/ Disallow: /sites/qlik/ Disallow: /whats-next/planes-trains-and-analytics Disallow: /top-50-wealth-managers Disallow: /toyota/fueled-by-skepticism Disallow: /toyota/coming-soon Disallow: /dell/cloud-future-of-medicine Disallow: /cit/renewable-energy Disallow: /retail Disallow: /mobile/cities-transformed Disallow: /cit/small-business-mergers-2 Disallow: /cit/small-business-mergers Disallow: /sites/kpmg/ Disallow: /dell/cities-transformed Disallow: /sites/neustar Disallow: /sites/ubs Disallow: /sites/northerntrust Disallow: /sites/infiniti Disallow: /sites/honeywell Disallow: /sites/charlesschwabadvisor Disallow: /sites/frisco Disallow: /sites/colehaan Disallow: /sites/gapincthiswayahead Disallow: /sites/discoverpersonalloans Disallow: /sites/delltechnologies Disallow: /colehaan/daring-to-step-forward Disallow: /johnhancock/ Disallow: /johnhancock/in-a-lifetime-healthcare Disallow: /johnhancock/in-a-lifetime-work Disallow: /oppenheimerfunds/the-upside Disallow: /sites/hpeenterprisenxt Disallow: /sites/rackspace Disallow: /sites/themacallan Disallow: /themacallan/worlds-together Disallow: /sites/inteliotpartnership Disallow: /oppenheimerfunds/the-upside Disallow: /partners/ Disallow: /johnhancock/in-a-lifetime-sustainability Disallow: /sites/domperignon Disallow: /sites/shinola Disallow: /sites/hpinc Disallow: /sites/hp Disallow: /sites/alfresco Disallow: /cbre/built-for-performance Disallow: /sites/mmedric Disallow: /sites/indiana Disallow: /sites/siemenshealthineers Disallow: /sites/rackspace Disallow: /rackspace/elevating-performance Disallow: /sites/suntrust Disallow: /sites/unicefusa Disallow: /sites/mitsubishiheavyindustries Disallow: /best-countries-for-business/list Disallow: /sites/paloaltonetworks Disallow: /sites/f5 Disallow: /sites/samsungtv Disallow: /sites/qa Disallow: /sites/paycom Disallow: /sites/testbrandvoice Disallow: /business/blahblah Disallow: /sites/spdr Disallow: /sites/lincoln Disallow: /sites/abb Disallow: /sites/eycybersecurity Disallow: /sites/civicnation Disallow: /sites/bcg Disallow: /sites/vanguard Disallow: /sites/akamai Disallow: /sites/kyocera Disallow: /sites/socialimpact Disallow: /sites/emirates Disallow: /sites/hpincemea Disallow: /sites/ibmperformancecomputing Disallow: /sites/pfizer Disallow: /sites/ibmpowersystems Disallow: /sites/vanguardcharitable Disallow: /sites/suntrustprivatewealth Disallow: /sites/capitalone Disallow: /sites/juniper Disallow: /sites/wellsfargo Disallow: /sites/bnymellon Disallow: /sites/braintree Disallow: /sites/bnymellonwealthmanagement Disallow: /sites/epson Disallow: /sites/thomsonreuters Disallow: /sites/paysafe Disallow: /sites/josabank Disallow: /sites/hennessyparadisimperial Disallow: /sites/cartier Disallow: /sites/anabluewing Disallow: /sites/eidoo Disallow: /sites/cmstesting Disallow: /sites/adobepartnership Disallow: /sites/cornelljohnson Disallow: /sites/quantcast Disallow: /sites/japan Disallow: /sites/impactpartners Disallow: /sites/nvidia Disallow: /sites/personalcapital Disallow: /sites/microsoft365 Disallow: /sites/providencedigitalinnovation Disallow: /sites/whittiertrust Disallow: /sites/kinetica Disallow: /sites/paulevans Disallow: /sites/ritz-carltonleadershipcenter Disallow: /sites/icoengine Disallow: /sites/tesaro Disallow: /sites/theglenlivet Disallow: /feature/sf-test Disallow: /sites/courtyardmarriott Disallow: /sites/nuveen Disallow: /sites/intelai Disallow: /sites/secret Disallow: /sites/hec Disallow: /retail/old Disallow: /sites/alliantgroup Disallow: /sites/intuitivesurgical Disallow: /sites/sunovion Disallow: /sites/ofx Disallow: /sites/firstrepublicbank Disallow: /sites/1850byfolgers Disallow: /sites/audi Disallow: /sites/nuveencef Disallow: /sites/1850brandcoffee Disallow: /sites/cisco Disallow: /sites/blackstarcapitalltd Disallow: /sites/usbankaltitudereserve Disallow: /sites/dassaultsystemes Disallow: /sites/urogen Disallow: /sites/smartsheet Disallow: /sites/iwc Disallow: /sites/datingcom Disallow: /sites/intuitquickbooks Disallow: /sites/oticon Disallow: /sites/servicenow Disallow: /sites/deloitteriskandfinancialadvisory Disallow: /sites/dbsprivatebank Disallow: /sites/americanexpress Disallow: /sites/tanotofoundation Disallow: /sites/citizensaccess Disallow: /sites/salesforceessentials Disallow: /sites/drexeluniversity Disallow: /sites/microsoftinnovation Disallow: /sites/mypos Disallow: /sites/harvardpilgrimhealthcare Disallow: /sites/appdynamics Disallow: /sites/ministryoftourismindia Disallow: /sites/orlando Disallow: /sites/extrahop Disallow: /sites/ascm Disallow: /sites/crowe Disallow: /sites/peerfit Disallow: /sites/theleukemialymphomasociety Disallow: /sites/hewlettpackardenterprise Disallow: /sites/montblanc Disallow: /sites/meero Disallow: /sites/braze Disallow: /sites/publicissapient Disallow: /sites/flexport User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: GPTBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: meta-externalagent User-agent: Meta-ExternalAgent Disallow: / User-Agent: OAI-SearchBot Allow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-Agent: PerplexityBot Disallow: / Sitemap: https://www.forbes.com/news_sitemap.xml Sitemap: https://www.forbes.com/video_sitemap.xml Sitemap: https://www.forbes.com/companies_sitemap.xml Sitemap: https://www.forbes.com/health/sitemap.xml Sitemap: https://www.forbes.com/connect/events-sitemap.xml Sitemap: https://www.forbes.com/advisor/sitemap.xml Sitemap: https://www.forbes.com/home-improvement/sitemap.xml