Free tools
Robots.txt Examples
Robots.txt file content for axios.com.
Robot.txt file for: axios.com
User-agent: Twitterbot/* Disallow: Crawl-delay: 1 User-agent: * Disallow: /core/* User-agent: GPTBot Disallow: / Sitemap: https://www.axios.com/sitemap.xml # 8fkux4mqab196bbs