Free tools
Robots.txt Examples
Robots.txt file content for ukr.net.
Robot.txt file for: ukr.net
User-agent: Mediapartners-Google Disallow: User-Agent: MJ12bot Disallow: User-agent: * Disallow: /*how_to_make_homepage/* Disallow: /news/*/page*.html Disallow: /menu.html Disallow: /search.html Disallow: /weather.html Disallow: /weather/* Disallow: /view/* Disallow: /currency.html Disallow: /currency/* Disallow: /fuels.html Disallow: /fuels/change_region.html* Disallow: /favorites.html Disallow: /market.html Disallow: /change_region.html Disallow: *?* Disallow:/views/main/* Disallow:/news/search/* Disallow:/news/subject.html Disallow:/book/rusnet.html Disallow:/news/politika/blogs.html Disallow:/book/vstuplenie.html Disallow:/weather/* Disallow:/dat/main/* User-agent: facebookexternalhit Disallow: User-agent: Twitterbot Disallow: Host: www.ukr.net Sitemap: https://www.ukr.net/sitemap.xml