Free tools

Robots.txt Examples

Robots.txt file content for pinkvilla.com.

Robot.txt file for: pinkvilla.com

      User-agent: *
Allow: /
Allow: *.css
Allow: *.woff
Allow: *.woff2
Allow: *.js
Disallow: /scripts/
Disallow: /video/*/fallback?noscript

# Paths (clean URLs)
Disallow: /admin/
Disallow: /logout/
Disallow: /search/

User-agent: Googlebot-News
Disallow: /rooms/*

User-agent: grapeshot 
Disallow:

Sitemap: https://www.pinkvilla.com/news-sitemap.xml
Sitemap: https://www.pinkvilla.com/image-sitemap-gen.xml
Sitemap: https://www.pinkvilla.com/sitemap.xml