Free tools

Robots.txt Examples

Robots.txt file content for kompas.com.

Robot.txt file for: kompas.com

      User-Agent: *
Disallow: /search/?q=
Disallow: /komentar/*
Disallow: /copy/*
Disallow: *?jxrecoid=*
Disallow: *?utm_source=*
Disallow: *?source=*

User-Agent: GPTBot
Disallow: /

Sitemap: https://www.kompas.com/sitemap.xml