Free tools

Robots.txt Examples

Robots.txt file content for superpages.com.

Robot.txt file for: superpages.com

      User-agent: Mediapartners-Google
Disallow:

User-Agent: *
Disallow: /*images/li.gif
Disallow: /*images/logging_requests.gif
Disallow: /relevance_feedback
Disallow: /listings/
Disallow: /listing_feedback/
Disallow: */report_abuse
Disallow: /gallery/*/copyright
Disallow: /gallery/*/flag
Disallow: /contribute/
Disallow: /reservations/
Disallow: */print_ad?*
Disallow: */audio_ad?*
Disallow: */map_locations
Disallow: /reviews/*/up
Disallow: /reviews/*/down
Disallow: /reviews/*/follow
Disallow: /reviews/*/unfollow
Disallow: /semp/*
Disallow: /login
Disallow: /register
Disallow: /user/
Disallow: /undefined/
Disallow: /lwes/
Disallow: /route?*

User-agent: 008
Disallow: /

User-agent: scrapy
Disallow: /

User-agent: Twitterbot
Allow: *

Host: www.superpages.com