Free tools

Robots.txt Examples

Robots.txt file content for olx.ua.

Robot.txt file for: olx.ua

      # sitecode:olxua-desktop
Host: https://www.olx.ua
Sitemap: https://www.olx.ua/sitemap.xml
User-agent: *
Disallow: */ajax/
Disallow: */adminpanel/
Disallow: */facebook/
Disallow: */rss/
Disallow: */account/
Disallow: */myaccount/
Disallow: */print/
Disallow: */chat/
Disallow: */i2/obyavlenie/abuse/
Disallow: */m/obyavlenie/abuse/
Disallow: */bundles/
Disallow: */obyavlenie/flyer/
Disallow: */obyavlenie/contact/
Disallow: */payment/
Disallow: */post-new-ad
Disallow: */favorites
Disallow: */abuse/
Disallow: */search/
Disallow: */observed
Disallow: */adding
Disallow: */list/next/
Disallow: */list/prev/
Disallow: */api/v1/users/me/
Disallow: */api/v1/offers/*/page-views/
Disallow: */api/v1/offers/*/suggested/
Disallow: */api/v1/targeting/
Disallow: */api/v1/offers/metadata/
Disallow: */i2/obyavlenie/contact/*
Disallow: /api/open/oauth/token/
Disallow: */i2/*
Allow: /