Free tools

Robots.txt Examples

Robots.txt file content for theknot.com.

Robot.txt file for: theknot.com

      User-agent: *
Disallow: /search*
Disallow: /dashboard
Disallow: /boards*
Disallow: /account/
Disallow: /inbox*
Disallow: /gs/dashboard
Disallow: /us/guest/pin*
Disallow:  /registry/ofc
Disallow: /registry/couplesearch/*
Disallow: /more/campaigns/*
Disallow: /fashion/boards/*
Disallow: /collaboration*
Disallow: */competitive_ads
Allow: /content/topic/amphtml/*
Disallow: /content/topic/*
Disallow: /tmp-landing-pages/*
Disallow: /paper/user/login*
Disallow: /paper/user/register*
Disallow: /paper/user/password*
Disallow: /paper/canvas/*
Disallow: /paper/order/*
Disallow: /paper/cart
Disallow: /paper/checkout/*
Disallow: /paper/your-orders
Disallow: /paper/your-drafts
 
User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-News
Allow: /news*
Disallow: /

User-agent: voltron
Disallow: /

User-agent: GPTBot
Disallow: /