Free tools

Robots.txt Examples

Robots.txt file content for wanderlog.com.

Robot.txt file for: wanderlog.com

      # Group 1
User-agent: *
Disallow: /plan/
Disallow: /plans/
Disallow: /friendsPlans/
Disallow: /history/
Disallow: /settings/
Disallow: /notifications/
Disallow: /extraReviews.js
Disallow: /placePageSnippets.js
Disallow: /refer/

# We used to disallow /api/places/redirectToFirstSearchResult/, but that leads to those urls being indexed
# Instead, we add the "X-Robots-Tag: noindex" header to the redirect response

Sitemap: https://wanderlog.com/sitemap.xml