Free tools
Robots.txt Examples
Robots.txt file content for stern.de.
Robot.txt file for: stern.de
User-agent: msnbot-media/1.1 Disallow: / User-agent: msnbot-media/2.0b Disallow: / User-agent: Googlebot-News Disallow: /noch-fragen/ Disallow: /action/ Disallow: /sn/ Disallow: /syndication/ Disallow: /elastic/ Disallow: /gutscheine/ Disallow: /search? Disallow: /vergleich/ Disallow: /p-api/* Disallow: /siegel/aerzte/ User-agent: scrapy Disallow: / User-agent: Diffbot Disallow: / User-agent: RyteBot Disallow: / User-agent: proximic Disallow: /*&share=* User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: /action/ Disallow: /sn/ Disallow: /syndication/ Disallow: /elastic/ Disallow: /gutscheine/wp-admin/ Disallow: /search? Disallow: /vergleich/zum/ Disallow: /siegel/aerzte/ Disallow: /p-api/ Disallow: /p-user/