Free tools

Robots.txt Examples

Robots.txt file content for univision.com.

Robot.txt file for: univision.com

      User-agent: *
Allow: /
Disallow: /deportes/fantasy/template*
Disallow: /noticias/live-fyre-blog-test*
Disallow: /verizon-connecta-la-banda*
Disallow: /temas/verizon-connecta-la-banda*
Disallow: /temas/test*
Disallow: /test*
Disallow: /amp/test*
Disallow: /en/*
Disallow: /buscar
Disallow: /6881
Disallow: /beta
Disallow: /*playlist.xml
Disallow: /_preview
Disallow: /search
Disallow: */embed*

User-agent: netseer
Disallow: /

User-agent: Clickagy Intelligence Bot v2
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: UbiCrawler
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /

User-agent: Maxthon
Disallow: /

User-agent: CNCDialer
Disallow: /

User Agent: Grapeshot
Do not allow: /

# Sitemaps and Autodiscovers
Sitemap: https://www.univision.com/sitemap.xml
Sitemap: https://www.univision.com/video-sitemap.xml
# Google News
Sitemap: https://www.univision.com/feed-sitemap/google-news
Sitemap: https://www.univision.com/feed-sitemap/google-news/famosos
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-arizona
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-austin
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-chicago
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-bakersfield
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-dallas
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-fresno
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-houston
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-los-angeles
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-miami
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-north-carolina
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-philadelphia
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-puerto-rico
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-sacramento
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-salt-lake-city
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-san-antonio
Sitemap: https://www.univision.com/feed-sitemap/google-news/local-san-francisco
Sitemap: https://www.univision.com/feed-sitemap/google-news/noticias
Sitemap: https://www.univision.com/feed-sitemap/google-news/elecciones-2020