Free tools

Robots.txt Examples

Robots.txt file content for axios.com.

Robot.txt file for: axios.com

      User-agent: Twitterbot/*
Disallow:
Crawl-delay: 1

User-agent: *
Disallow: /core/*

User-agent: GPTBot
Disallow: /

Sitemap: https://www.axios.com/sitemap.xml

# 8fkux4mqab196bbs