Free tools
Robots.txt Examples
Robots.txt file content for bigl.ua.
Robot.txt file for: bigl.ua
User-Agent: * Disallow: /*? Disallow: /search Disallow: /redirect Disallow: /debug/ Disallow: /iframe.html Disallow: /remote/ Disallow: /catalog/ Disallow: /js/ckeditor_latest/ Disallow: /opensearch.xml Disallow: /R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw== User-Agent: YadirectBot Disallow: / User-Agent: YandexDirect Disallow: / User-agent: Yandex Disallow: / User-Agent: YandexBot Disallow: / User-agent: Yahoo Disallow: / User-agent: MJ12bot Disallow: / User-agent: baidu Disallow: / User-agent: dotbot Disallow: / User-agent: grapeshot Disallow: / User-agent: oBot Disallow: / User-agent: UptimeRobot Disallow: / User-agent: Wget Disallow: / User-agent: Scrapy Disallow: / User-agent: Twitterbot Disallow: / User-agent: GuzzleHttp Disallow: / User-agent: X11 Disallow: / User-agent: SemrushBot Disallow: / User-agent: Rambler Disallow: / User-agent: Linguee Bot Disallow: / User-agent: Webcrawler Disallow: / User-agent: proximic Disallow: / User-agent: Trident Disallow: / User-agent: bingbot Disallow: / Host: https://bigl.ua Sitemap: https://bigl.ua/ua/sitemap.xml