Free tools

Robots.txt Examples

Robots.txt file content for bigl.ua.

Robot.txt file for: bigl.ua

      User-Agent: *
Disallow: /*?
Disallow: /search
Disallow: /redirect
Disallow: /debug/
Disallow: /iframe.html
Disallow: /remote/
Disallow: /catalog/
Disallow: /js/ckeditor_latest/
Disallow: /opensearch.xml
Disallow: /R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==

User-Agent: YadirectBot
Disallow: /

User-Agent: YandexDirect
Disallow: /

User-agent: Yandex
Disallow: /

User-Agent: YandexBot
Disallow: /

User-agent: Yahoo
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: baidu
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: grapeshot
Disallow: /

User-agent: oBot
Disallow: /

User-agent: UptimeRobot
Disallow: /

User-agent: Wget
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Twitterbot
Disallow: /

User-agent: GuzzleHttp
Disallow: /

User-agent: X11
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: Rambler
Disallow: /

User-agent: Linguee Bot
Disallow: /

User-agent: Webcrawler
Disallow: /

User-agent: proximic
Disallow: /

User-agent: Trident
Disallow: /

User-agent: bingbot
Disallow: /

Host: https://bigl.ua
Sitemap: https://bigl.ua/ua/sitemap.xml