Free tools
Robots.txt Examples
Robots.txt file content for pixiv.net.
Robot.txt file for: pixiv.net
User-agent: * Disallow: /cdn-cgi/ Disallow: /rpc/index.php?mode=profile_module_illusts&user_id=*&illust_id=* Disallow: /ajax/illust/*/recommend/init Disallow: *return_to* Disallow: /?return_to= Disallow: /login.php?return_to= Disallow: /index.php?return_to= Disallow: /artworks/unlisted/* Disallow: /users/*/followers Disallow: /users/*/mypixiv Disallow: /users/*/bookmarks Disallow: /novel/comments.php?id= Disallow: /novels/unlisted/* Disallow: /en/group Disallow: /en/search/ Disallow: /en/users/*/followers Disallow: /en/users/*/mypixiv Disallow: /en/users/*/bookmarks Disallow: /en/novel/comments.php?id= Disallow: /fanbox/search Disallow: /fanbox/tag Allow: /comic-indies/$ Allow: /comic-indies/about Disallow: /comic-indies/