Free tools

Robots.txt Examples

Robots.txt file content for deviantart.com.

Robot.txt file for: deviantart.com

      User-agent: *
Disallow: /users/*?
Disallow: /join/*?
Disallow: /morelikethis/
Disallow: /download/
Disallow: /checkout/
Disallow: /global/
Disallow: /api/
Disallow: /critiques/
Disallow: /prints/*by:*
Disallow: /_stash/
Crawl-delay: 1

User-agent: Pinterestbot
Disallow: /
 
Sitemap: https://www.deviantart.com/sitemap-index.xml.gz