Free tools

Robots.txt Examples

Robots.txt file content for dw.com.

Robot.txt file for: dw.com

      User-agent: * 
Disallow: /search/
Disallow: /overlay/
Disallow: /popups/mediaplayer/
Disallow: /popups/popup_gallery/
Disallow: /*/layoutvorlagen/
Disallow: /*/user/account$
Disallow: /*/user/activity$
Disallow: /*/user/profile$
Disallow: /*/user/password/change$
Disallow: /*/user/password/set$
Disallow: /*/user/feedback/status?type=*
Disallow: /*/user/register/confirm$
Disallow: /*/user/email/change$
Disallow: /*?maca=*

User-agent: Twitterbot
Allow: /*?maca=*

Sitemap: https://www.dw.com/sitemap.xml