Free tools
Robots.txt Examples
Robots.txt file content for superpages.com.
Robot.txt file for: superpages.com
User-agent: Mediapartners-Google Disallow: User-Agent: * Disallow: /*images/li.gif Disallow: /*images/logging_requests.gif Disallow: /relevance_feedback Disallow: /listings/ Disallow: /listing_feedback/ Disallow: */report_abuse Disallow: /gallery/*/copyright Disallow: /gallery/*/flag Disallow: /contribute/ Disallow: /reservations/ Disallow: */print_ad?* Disallow: */audio_ad?* Disallow: */map_locations Disallow: /reviews/*/up Disallow: /reviews/*/down Disallow: /reviews/*/follow Disallow: /reviews/*/unfollow Disallow: /semp/* Disallow: /login Disallow: /register Disallow: /user/ Disallow: /undefined/ Disallow: /lwes/ Disallow: /route?* User-agent: 008 Disallow: / User-agent: scrapy Disallow: / User-agent: Twitterbot Allow: * Host: www.superpages.com