Free tools

Robots.txt Examples

Robots.txt file content for 123rf.com.

Robot.txt file for: 123rf.com

      User-agent: *
Disallow: /downloadcomp.php
Disallow: /profileinterview.php
Disallow: /pdf_lightbox.php
Disallow: /downloadpubliclightbox.php
Disallow: /mylightbox.php
Disallow: /similar.php
Disallow: /similarmodel_ajax.php
Disallow: /bova/
Disallow: /images/verify_image.png
Disallow: /images/welcome_image.png
Disallow: /images/ga_img_handler.php
Disallow: /bigdata/logging/
Disallow: /visual/

User-agent: GPTBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /