Free tools

Robots.txt Examples

Robots.txt file content for ameblo.jp.

Robot.txt file for: ameblo.jp

      User-agent: Googlebot
Disallow: /*/page-*.html
Allow: /*/entrylist.html
Allow: /*/entrylist-*.html
Allow: /*/entry-*.html
Disallow: /*/entry*-*
Disallow: /*/day*.html
Disallow: /*/reader.html
Disallow: /*/reader-*.html
Disallow: /*/favorite.html
Disallow: /*/favorite-*.html
Disallow: /*/bookmark.html
Disallow: /*/bookmark-*.html
Disallow: /*/amemberentrylist.html
Disallow: /*/amemberentrylist-*.html
Disallow: /*/amemberentry-*.html
Disallow: /*/imagelist.html
Disallow: /*/imagelist-*.html
Disallow: /*/image-*-*.html
Allow: /nonstyleishida/videolist.html
Allow: /nonstyleishida/videolist-*.html
Allow: /inouenonstyle/videolist.html
Allow: /inouenonstyle/videolist-*.html
Allow: /chiaki1007akie/videolist.html
Allow: /chiaki1007akie/videolist-*.html
Allow: /takaoka-saki-blog/videolist.html
Allow: /takaoka-saki-blog/videolist-*.html
Allow: /sakuamema/videolist.html
Allow: /sakuamema/videolist-*.html
Allow: /norokayoblog/videolist.html
Allow: /norokayoblog/videolist-*.html
Allow: /o-rukyojin1116/videolist.html
Allow: /o-rukyojin1116/videolist-*.html
Allow: /kandaseiko1030/videolist.html
Allow: /kandaseiko1030/videolist-*.html
Disallow: /*/videolist.html
Disallow: /*/videolist-*.html
Allow: /*/video-*.html$
Disallow: /*/video-*.html?*vnm=*
Disallow: /*/message-board.html
Disallow: /*/comment-*
Disallow: /*/themetop.html
Disallow: /*/iine_entry-*.html
Disallow: /summary/api/
Disallow: /accesslog/BlogAccessLog
Disallow: /p/
Disallow: /_api$
Disallow: /_api/
Disallow: /_hook/*.html
Disallow: /blog_portal/category/*/news
Disallow: /_static/
Disallow: /s/embed/reblog-card/

User-agent: bingbot
Crawl-delay: 1