Free tools

Robots.txt Examples

Robots.txt file content for theglobeandmail.com.

Robot.txt file for: theglobeandmail.com

      User-agent: Googlebot-News
Disallow: /feeds/
Disallow: /incoming/
Disallow: /test/
Disallow: /partners/
Disallow: /search/
Disallow: /business/adv/appointmentnotices/search/

User-agent: AdsBot-Google
Disallow: /feeds/
Disallow: /incoming/
Disallow: /test/
Disallow: /search/
Disallow: /business/adv/appointmentnotices/search/

User-agent: *
Disallow: /feeds/
Disallow: /incoming/
Disallow: /test/
Disallow: /search/
Disallow: /business/adv/appointmentnotices/search/
Disallow: /marketing-containers/
Disallow: /coupons/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.theglobeandmail.com/sitemap.xml?outputType=xml