Free tools
Robots.txt Examples
Robots.txt file content for theknot.com.
Robot.txt file for: theknot.com
User-agent: * Disallow: /search* Disallow: /dashboard Disallow: /boards* Disallow: /account/ Disallow: /inbox* Disallow: /gs/dashboard Disallow: /us/guest/pin* Disallow: /registry/ofc Disallow: /registry/couplesearch/* Disallow: /more/campaigns/* Disallow: /fashion/boards/* Disallow: /collaboration* Disallow: */competitive_ads Allow: /content/topic/amphtml/* Disallow: /content/topic/* Disallow: /tmp-landing-pages/* Disallow: /paper/user/login* Disallow: /paper/user/register* Disallow: /paper/user/password* Disallow: /paper/canvas/* Disallow: /paper/order/* Disallow: /paper/cart Disallow: /paper/checkout/* Disallow: /paper/your-orders Disallow: /paper/your-drafts User-agent: Mediapartners-Google Allow: / User-agent: Googlebot-News Allow: /news* Disallow: / User-agent: voltron Disallow: / User-agent: GPTBot Disallow: /