Free tools
Robots.txt Examples
Robots.txt file content for spanishdict.com.
Robot.txt file for: spanishdict.com
User-agent: * Disallow: /m/translate/ Allow: /users/login Allow: /users/signup_start Disallow: /drill Disallow: /users/ Disallow: /answers/dictionary_answers_search/ Disallow: /dictionary/phrasebook_search/ Disallow: /dictionary/answers_search/ Disallow: /dictionary/translation_sdl Disallow: /dictionary/translation_prompt Disallow: /dictionary/translation_ms Disallow: /flashcards/print_view Disallow: /flashcards/json Disallow: /vocabulary/choose-level Disallow: /vocabulario/escoge-tu-nivel Disallow: /examples/la%2520periodista%2520escribe%2520un%2520mensaje%2520electronico%3Ft%3D1 Allow: / Sitemap: https://www.spanishdict.com/sitemap-index-https.xml User-agent: CriteoBot/0.1 Crawl-delay: 1 Disallow: /m/translate/ Allow: /users/login Allow: /users/signup_start Disallow: /drill Disallow: /users/ Disallow: /answers/dictionary_answers_search/ Disallow: /dictionary/phrasebook_search/ Disallow: /dictionary/answers_search/ Disallow: /dictionary/translation_sdl Disallow: /dictionary/translation_prompt Disallow: /dictionary/translation_ms Disallow: /flashcards/print_view Disallow: /flashcards/json Disallow: /vocabulary/choose-level Disallow: /vocabulario/escoge-tu-nivel Disallow: /examples/la%2520periodista%2520escribe%2520un%2520mensaje%2520electronico%3Ft%3D1 Allow: / Sitemap: https://www.spanishdict.com/sitemap-index-https.xml User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Applebot-Extended Disallow: /