Free tools
Robots.txt Examples
Robots.txt file content for dbpedia.org.
Robot.txt file for: dbpedia.org
User-agent: * Crawl-delay: 10 Disallow: /admin Disallow: /tutorial Disallow: /eNews Disallow: /IBuySpy Disallow: /PortalCS Disallow: /jdbcdemo Disallow: /forums Disallow: /xslt Disallow: /webapp Disallow: /xqdemo Disallow: /HtmlPivotViewer Disallow: /DeepZoomCache Disallow: /fct