Free tools

Robots.txt Examples

Robots.txt file content for behance.net.

Robot.txt file for: behance.net

      Sitemap: https://www.behance.net/sitemap

# Disallow AI bots
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCbot
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Diffbot
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ImagesiftBot
User-agent: Meta-ExternalAgent
User-agent: Omgili
User-agent: Omgilibot
User-agent: Timpibot
User-agent: Webzio-Extended
Disallow: /

User-agent: *
Disallow: /v2/
Disallow: /v3/
Disallow: /log$
Disallow: /search/*?*assetsCategory=*
Disallow: /search/*?*assetsPriceTier=*
Disallow: /search/*?*assetsExtensionTypes=*
Disallow: /search/*?*color_hex=*
Disallow: /search/*?*country=*
Disallow: /search/*?*exif_aperture=*
Disallow: /search/*?*exif_focal_length=*
Disallow: /search/*?*exif_lens_brand=*
Disallow: /search/*?*exif_make=*
Disallow: /search/*?*exif_shutter_speed=*
Disallow: /search/*?*exif_zoom=*
Disallow: /search/*?*field=*
Disallow: /search/*?*gif=*
Disallow: /search/*?*premium=*
Disallow: /search/*?*sort=*
Disallow: /search/*?*schools=*
Disallow: /search/*?*tools=*
Disallow: /search/*?*time=*
Disallow: /search/*?*userAvailability=*
Disallow: /hire/browse