Free tools
Robots.txt Examples
Robots.txt file content for rakuten.co.jp.
Robot.txt file for: rakuten.co.jp
User-Agent: * Disallow: /images/ Disallow: /backup/ Disallow: /cgi-bin/ Disallow: /shops/ Disallow: /shop/ascosing/ Disallow: /shop/egs-g/ Disallow: /shop/tripleo-shop/ Disallow: /shop/fooody/ Disallow: /shop/kira-con02/ Disallow: /*/lightbox_*.html Disallow: /aboutus/am/maint/