Free tools
Robots.txt Examples
Robots.txt file content for infobae.com.
Robot.txt file for: infobae.com
User-agent: * Sitemap: https://www.infobae.com/arc/outboundfeeds/sitemap2/ Sitemap: https://www.infobae.com/arc/outboundfeeds/news-sitemap2/ Sitemap: https://www.infobae.com/arc/outboundfeeds/sitemap2/category/america/ Sitemap: https://www.infobae.com/arc/outboundfeeds/news-sitemap2/category/america/ Sitemap: https://www.infobae.com/arc/outboundfeeds/sitemap2/category/teleshow/ Sitemap: https://www.infobae.com/arc/outboundfeeds/news-sitemap2/category/teleshow/ Sitemap: https://www.infobae.com/arc/outboundfeeds/sitemap-index-static/