Free tools

Robots.txt Examples

Robots.txt file content for spanishdict.com.

Robot.txt file for: spanishdict.com

      User-agent: *
Disallow: /m/translate/
Allow: /users/login
Allow: /users/signup_start
Disallow: /drill
Disallow: /users/
Disallow: /answers/dictionary_answers_search/
Disallow: /dictionary/phrasebook_search/
Disallow: /dictionary/answers_search/
Disallow: /dictionary/translation_sdl
Disallow: /dictionary/translation_prompt
Disallow: /dictionary/translation_ms
Disallow: /flashcards/print_view
Disallow: /flashcards/json
Disallow: /vocabulary/choose-level
Disallow: /vocabulario/escoge-tu-nivel
Disallow: /examples/la%2520periodista%2520escribe%2520un%2520mensaje%2520electronico%3Ft%3D1
Allow: /

Sitemap: https://www.spanishdict.com/sitemap-index-https.xml

User-agent: CriteoBot/0.1
Crawl-delay: 1
Disallow: /m/translate/
Allow: /users/login
Allow: /users/signup_start
Disallow: /drill
Disallow: /users/
Disallow: /answers/dictionary_answers_search/
Disallow: /dictionary/phrasebook_search/
Disallow: /dictionary/answers_search/
Disallow: /dictionary/translation_sdl
Disallow: /dictionary/translation_prompt
Disallow: /dictionary/translation_ms
Disallow: /flashcards/print_view
Disallow: /flashcards/json
Disallow: /vocabulary/choose-level
Disallow: /vocabulario/escoge-tu-nivel
Disallow: /examples/la%2520periodista%2520escribe%2520un%2520mensaje%2520electronico%3Ft%3D1
Allow: /

Sitemap: https://www.spanishdict.com/sitemap-index-https.xml

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /