Free tools
Robots.txt Examples
Robots.txt file content for univision.com.
Robot.txt file for: univision.com
User-agent: * Allow: / Disallow: /deportes/fantasy/template* Disallow: /noticias/live-fyre-blog-test* Disallow: /verizon-connecta-la-banda* Disallow: /temas/verizon-connecta-la-banda* Disallow: /temas/test* Disallow: /test* Disallow: /amp/test* Disallow: /en/* Disallow: /buscar Disallow: /6881 Disallow: /beta Disallow: /*playlist.xml Disallow: /_preview Disallow: /search Disallow: */embed* User-agent: netseer Disallow: / User-agent: Clickagy Intelligence Bot v2 Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: Maxthon Disallow: / User-agent: CNCDialer Disallow: / User Agent: Grapeshot Do not allow: / # Sitemaps and Autodiscovers Sitemap: https://www.univision.com/sitemap.xml Sitemap: https://www.univision.com/video-sitemap.xml # Google News Sitemap: https://www.univision.com/feed-sitemap/google-news Sitemap: https://www.univision.com/feed-sitemap/google-news/famosos Sitemap: https://www.univision.com/feed-sitemap/google-news/local-arizona Sitemap: https://www.univision.com/feed-sitemap/google-news/local-austin Sitemap: https://www.univision.com/feed-sitemap/google-news/local-chicago Sitemap: https://www.univision.com/feed-sitemap/google-news/local-bakersfield Sitemap: https://www.univision.com/feed-sitemap/google-news/local-dallas Sitemap: https://www.univision.com/feed-sitemap/google-news/local-fresno Sitemap: https://www.univision.com/feed-sitemap/google-news/local-houston Sitemap: https://www.univision.com/feed-sitemap/google-news/local-los-angeles Sitemap: https://www.univision.com/feed-sitemap/google-news/local-miami Sitemap: https://www.univision.com/feed-sitemap/google-news/local-north-carolina Sitemap: https://www.univision.com/feed-sitemap/google-news/local-philadelphia Sitemap: https://www.univision.com/feed-sitemap/google-news/local-puerto-rico Sitemap: https://www.univision.com/feed-sitemap/google-news/local-sacramento Sitemap: https://www.univision.com/feed-sitemap/google-news/local-salt-lake-city Sitemap: https://www.univision.com/feed-sitemap/google-news/local-san-antonio Sitemap: https://www.univision.com/feed-sitemap/google-news/local-san-francisco Sitemap: https://www.univision.com/feed-sitemap/google-news/noticias Sitemap: https://www.univision.com/feed-sitemap/google-news/elecciones-2020