Free tools

Robots.txt Examples

Robots.txt file content for stanford.edu.

Robot.txt file for: stanford.edu

      User-agent: *
Disallow: /~folutade/ATI/
Disallow: /~neubank/Eubank_EthnicityNetworksPaper.pdf
Disallow: /~neubank/Eubank_EthnicityNetworksPaper2016.pdf
Disallow: /dept/soc/people/rsandefur/