Free tools
Robots.txt Examples
Robots.txt file content for justanswer.com.
Robot.txt file for: justanswer.com
User-agent: Xombot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Zing-BottaBot User-agent: Zing-BottaBot/2.0 Disallow: / User-agent: GoScraper Disallow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: dotbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: trendictionbot Disallow: / User-agent: Barkrowler Disallow: / User-agent: Turnitin Disallow: / User-agent: MojeekBot Disallow: / User-agent: BingBot # Directories Disallow: /App_Controls/ Disallow: /LS/ Disallow: /misc/ Disallow: /modules/ Disallow: /profiles/ Disallow: /sites/ Disallow: /themes/ Disallow: /img/$ # Files Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /install.php Disallow: /INSTALL.txt Disallow: /LICENSE.txt Disallow: /MAINTAINERS.txt Disallow: /update.php Disallow: /UPGRADE.txt Disallow: /xmlrpc.php # Paths (clean URLs) Disallow: /de/ Disallow: /ja/ Disallow: /es/ Disallow: /en-gb/ Disallow: /admin/ Disallow: /archives/ Disallow: /comment/reply/ Disallow: /contact/ Disallow: /fling/ Disallow: /fraud-examiner/ Disallow: /ja_services/ Disallow: /logout/ Disallow: /node/add/ Disallow: /processes/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ # Paths (no clean URLs) Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=contact/ Disallow: /?q=logout/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /boost_stats.php Disallow: /disclaimer.html # Secure and Error pages Disallow: /account/ Disallow: /billing/ Disallow: /payment/ Disallow: /new-question/ Disallow: /email_notify_confirm.aspx Disallow: /errorfile500.aspx Disallow: /expert/info_submit.aspx Disallow: /insufficient-permission.aspx Disallow: /my-questions.aspx Disallow: /pm/new_message.aspx Disallow: /question/accepted-question.aspx Disallow: /question/disclaimer.aspx Disallow: /question/qa.aspx Disallow: /ratings.aspx Disallow: /report_post.aspx Disallow: /unsubscribe.aspx Disallow: /login.aspx Disallow: /feedback/leave-feedback.aspx Disallow: /register.aspx # 404s Disallow: /consumer%20electronics/? Disallow: /consumer%20protection? Disallow: /classic-cars/classic%20corvette? Disallow: /australia-law? Disallow: /australia-law/? Disallow: /anwalt/? Disallow: /medizin/? Disallow: /law/l? Disallow: /law/r? Disallow: /cash-registers/cash-register? Disallow: /electronics/cash-register/cash-register? Disallow: /electronics/printers-and-office/Verifone/? Disallow: /boat-mechanics? Disallow: /car-mechanics? Disallow: /mental-health/counselors/? Disallow: /homework/writing? Disallow: /homework/math? Disallow: /homework/math/college-math-homework? Disallow: /homework/math/advanced-math-homework? # Sitemaps Sitemap: https://www.justanswer.com/sitemap.xml # Exceptions Allow: /processes/bundle/BundleContent.ashx Allow: /processes/css.ashx Allow: /processes/js.ashx Allow: /processes/funnel/VirtualAssistantService.asmx/GetAssistantProfile Allow: /processes/funnel/assistant-profile.ashx Allow: /processes/expert/expert-feedback/ExpertFeedbackStatisticsService.asmx/GetSpecificFeedbackStatistics User-agent: * Crawl-delay: 1 # Directories Disallow: /App_Controls/ Disallow: /LS/ Disallow: /misc/ Disallow: /modules/ Disallow: /profiles/ Disallow: /sites/ Disallow: /themes/ Disallow: /img/$ # Files Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /install.php Disallow: /INSTALL.txt Disallow: /LICENSE.txt Disallow: /MAINTAINERS.txt Disallow: /update.php Disallow: /UPGRADE.txt Disallow: /xmlrpc.php # Paths (clean URLs) Disallow: /de/ Disallow: /ja/ Disallow: /es/ Disallow: /en-gb/ Disallow: /admin/ Disallow: /archives/ Disallow: /comment/reply/ Disallow: /contact/ Disallow: /fling/ Disallow: /fraud-examiner/ Disallow: /ja_services/ Disallow: /logout/ Disallow: /node/add/ Disallow: /processes/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ # Paths (no clean URLs) Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=contact/ Disallow: /?q=logout/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /boost_stats.php Disallow: /disclaimer.html # Secure and Error pages Disallow: /account/ Disallow: /billing/ Disallow: /payment/ Disallow: /new-question/ Disallow: /email_notify_confirm.aspx Disallow: /errorfile500.aspx Disallow: /expert/info_submit.aspx Disallow: /insufficient-permission.aspx Disallow: /my-questions.aspx Disallow: /pm/new_message.aspx Disallow: /question/accepted-question.aspx Disallow: /question/disclaimer.aspx Disallow: /question/qa.aspx Disallow: /ratings.aspx Disallow: /report_post.aspx Disallow: /unsubscribe.aspx Disallow: /login.aspx Disallow: /feedback/leave-feedback.aspx Disallow: /register.aspx # 404s Disallow: /consumer%20electronics/? Disallow: /consumer%20protection? Disallow: /classic-cars/classic%20corvette? Disallow: /australia-law? Disallow: /australia-law/? Disallow: /anwalt/? Disallow: /medizin/? Disallow: /law/l? Disallow: /law/r? Disallow: /cash-registers/cash-register? Disallow: /electronics/cash-register/cash-register? Disallow: /electronics/printers-and-office/Verifone/? Disallow: /boat-mechanics? Disallow: /car-mechanics? Disallow: /mental-health/counselors/? Disallow: /homework/writing? Disallow: /homework/math? Disallow: /homework/math/college-math-homework? Disallow: /homework/math/advanced-math-homework? # Sitemaps Sitemap: https://www.justanswer.com/sitemap.xml # Exceptions Allow: /processes/bundle/BundleContent.ashx Allow: /processes/css.ashx Allow: /processes/js.ashx Allow: /processes/funnel/VirtualAssistantService.asmx/GetAssistantProfile Allow: /processes/funnel/assistant-profile.ashx Allow: /processes/expert/expert-feedback/ExpertFeedbackStatisticsService.asmx/GetSpecificFeedbackStatistics