Free tools
Robots.txt Generator
Generate, customize robots.txt file for your website for free. Robots.txt files are used to instruct web crawlers and other web robots how to crawl your website.
Generate, customize robots.txt file for your website for free. Robots.txt files are used to instruct web crawlers and other web robots how to crawl your website.
To add sitemap to robots txt is a good practice and this is how you add sitemap :
Sitemap: / <folder name> /