Free tools

Robots.txt Examples

Robots.txt file content for ludwig.guru.

Robot.txt file for: ludwig.guru

      User-Agent: *
Disallow: /s2
Disallow: /s2/*
Disallow: /it/s2
Disallow: /it/s2/*
Disallow: /en/s2
Disallow: /en/s2/*
Disallow: /zh/s2
Disallow: /zh/s2/*
Disallow: /account
Disallow: /account/*
Disallow: /it/account
Disallow: /zh/account
Disallow: /it/account/*
Disallow: /zh/account/*
Disallow: /dictionary/*
Disallow: /it/dictionary/*
Disallow: /zh/dictionary/*
Disallow: /share
Disallow: /share/*
Disallow: /it/share
Disallow: /zh/share
Disallow: /it/share/*
Disallow: /zh/share/*
Disallow: /login/limit
Disallow: /login/limit/*
Disallow: /it/login/limit
Disallow: /zh/login/limit
Disallow: /it/login/limit/*
Disallow: /zh/login/limit/*

User-Agent: Googlebot
Allow: /
Disallow: /account
Disallow: /account/*
Disallow: /it/account
Disallow: /zh/account
Disallow: /it/account/*
Disallow: /zh/account/*
Disallow: /dictionary/*
Disallow: /it/dictionary/*
Disallow: /zh/dictionary/*
Disallow: /share
Disallow: /share/*
Disallow: /it/share
Disallow: /zh/share
Disallow: /it/share/*
Disallow: /zh/share/*
Disallow: /login/limit
Disallow: /login/limit/*
Disallow: /it/login/limit
Disallow: /zh/login/limit
Disallow: /it/login/limit/*
Disallow: /zh/login/limit/*

User-agent: Yandex
Disallow: /dictionary/*
Disallow: /it/dictionary/*
Disallow: /zh/dictionary/*
Disallow: /account
Disallow: /account/*
Disallow: /it/account
Disallow: /zh/account
Disallow: /it/account/*
Disallow: /zh/account/*
Disallow: /share
Disallow: /share/*
Disallow: /it/share
Disallow: /zh/share
Disallow: /it/share/*
Disallow: /zh/share/*
Disallow: /login/limit
Disallow: /login/limit/*
Disallow: /it/login/limit
Disallow: /zh/login/limit
Disallow: /it/login/limit/*
Disallow: /zh/login/limit/*

User-agent: Yandex
Crawl-delay: 20

User-agent: grapeshot
Disallow: /

User-agent: proximic
Disallow: /

User-agent: Web-Crawler
Disallow: /

User-agent: bingbot
Crawl-delay: 20

User-agent: Bingbot
Crawl-delay: 20