Free tools

Robots.txt Examples

Robots.txt file content for tiktok.com.

Robot.txt file for: tiktok.com

      User-agent: Baiduspider
User-agent: 360Spider
User-agent: Sogouspider
User-agent: Yisouspider
User-agent: PetalBot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /foryou
Allow: /discover
Allow: /about
Allow: /legal
Allow: /safety
Allow: /forgood
Allow: /community-guidelines
Allow: /tag
Allow: /amp
Allow: /transparency
Allow: /share
Allow: /music
Disallow: /inapp
Disallow: /auth
Disallow: /embed/@
Disallow: /embed/v2
Disallow: /embed/curated
Disallow: /link
Disallow: */directory/
Disallow: /search/video?
Disallow: /search/user?q=
Disallow: /shop/view/product/

User-agent: Bingbot
Disallow: /discover