Free tools
Robots.txt Examples
Robots.txt file content for spiceworks.com.
Robot.txt file for: spiceworks.com
User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: yahoo-mmcrawler Disallow: User-agent: Twitterbot/1.0 Disallow: User-agent: TwitterBot 1.0 Disallow: User-agent: AhrefsBot Disallow: User-agent: AhrefsSiteAudit Disallow: User-agent: Screaming Frog Disallow: User-agent: Screaming Frog SEO Spider Disallow: User-agent: yahoo-blogs/v3.9 Disallow: User-agent: Bingbot Disallow: User-agent: * Disallow: / Disallow: /cgi-bin/ Disallow: /research/ Sitemap: https://www.spiceworks.com/sitemap_index.xml Disallow: /*? Disallow: /sw-marketing/* User-agent: * Disallow: /legal-resource-center/* User-agent: GPTBot Disallow: /