Free tools
Robots.txt Examples
Robots.txt file content for yellowpages.com.
Robot.txt file for: yellowpages.com
User-Agent: * Disallow: /*images/li.gif Disallow: /*images/logging_requests.gif Disallow: /relevance_feedback Disallow: /listings/ Disallow: /listing_feedback/ Disallow: */report_abuse Disallow: /gallery/*/copyright Disallow: /gallery/*/flag Disallow: /contribute/ Disallow: /reservations/ Disallow: */print_ad?* Disallow: */audio_ad?* Disallow: */map_locations Disallow: /reviews/*/up Disallow: /reviews/*/down Disallow: /reviews/*/follow Disallow: /reviews/*/unfollow Disallow: */no-internet-heading-assigned Disallow: */no-internet-heading-assisted Disallow: /login Disallow: /register Disallow: /user/ Disallow: /ypu/js/compiled/tripadvisor* Disallow: /ypu/apps/ypm-core/ypm/javascripts/bundle_tripadvisor* Disallow: /undefined/ Disallow: /improve_listing/* Disallow: /search* Disallow: /lwes/ Disallow: /route?* Disallow: /listings/*/directions* User-agent: 008 Disallow: / User-agent: scrapy Disallow: / User-agent: PiplBot Disallow: / User-agent: Twitterbot Allow: * Host: www.yellowpages.com