Free tools
Robots.txt Examples
Robots.txt file content for wanderlog.com.
Robot.txt file for: wanderlog.com
# Group 1 User-agent: * Disallow: /plan/ Disallow: /plans/ Disallow: /friendsPlans/ Disallow: /history/ Disallow: /settings/ Disallow: /notifications/ Disallow: /extraReviews.js Disallow: /placePageSnippets.js Disallow: /refer/ # We used to disallow /api/places/redirectToFirstSearchResult/, but that leads to those urls being indexed # Instead, we add the "X-Robots-Tag: noindex" header to the redirect response Sitemap: https://wanderlog.com/sitemap.xml