Free tools

Robots.txt Examples

Robots.txt file content for ukr.net.

Robot.txt file for: ukr.net

      User-agent: Mediapartners-Google
Disallow:

User-Agent: MJ12bot
Disallow: 

User-agent: *
Disallow: /*how_to_make_homepage/*
Disallow: /news/*/page*.html
Disallow: /menu.html
Disallow: /search.html
Disallow: /weather.html
Disallow: /weather/*
Disallow: /view/*
Disallow: /currency.html
Disallow: /currency/*
Disallow: /fuels.html
Disallow: /fuels/change_region.html*
Disallow: /favorites.html 
Disallow: /market.html
Disallow: /change_region.html 
Disallow: *?*
Disallow:/views/main/*
Disallow:/news/search/*
Disallow:/news/subject.html
Disallow:/book/rusnet.html
Disallow:/news/politika/blogs.html
Disallow:/book/vstuplenie.html
Disallow:/weather/*
Disallow:/dat/main/*

User-agent: facebookexternalhit
Disallow:

User-agent: Twitterbot
Disallow:

Host: www.ukr.net
Sitemap: https://www.ukr.net/sitemap.xml