Free tools

Robots.txt Examples

Robots.txt file content for digitaltrends.com.

Robot.txt file for: digitaltrends.com

      User-agent: magpie-crawler
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Disallow: /feed/

User-agent:*
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /*?s=
Disallow: /search
Disallow: /?search?
Disallow: /*?iab_category
Disallow: /*?product_tag=
Disallow: /*?dt_commerce_tag=
Disallow: /*?brand=
Disallow: /*?event=
Disallow: /*?product=
Disallow: /*?locations=
Disallow: /*?carrier=
Disallow: /*?entity=
Disallow: /*?game_platform=
Disallow: /*?persons=
Disallow: /*?streaming_service=
Disallow: /*?technology=
Disallow: /*?original_series=
Disallow: /tag/
Disallow: /brc/
Disallow: /sponsored/
Disallow: /aop/
Disallow: /aop-uk/
Disallow: /trash/
Disallow: /uncategorized/
Sitemap: https://www.digitaltrends.com/sitemap-all-articles-index.xml
Sitemap: https://www.digitaltrends.com/sitemap-wordle-answer-index.xml
Sitemap: https://www.digitaltrends.com/sitemap-all-images-index.xml
Sitemap: https://www.digitaltrends.com/sitemap-google-news-sitemap-index.xml