Free tools
Robots.txt Examples
Robots.txt file content for delish.com.
Robot.txt file for: delish.com
User-agent: Mediapartners-Google Disallow: User-agent: Facebot Disallow: User-agent: Twitterbot Disallow: User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Disallow: /au/ Disallow: /cn/ Disallow: /dk/ Disallow: /en/ Disallow: /es/ Disallow: /in/ Disallow: /it/ Disallow: /jp/ Disallow: /ng/ Disallow: /nl/ Disallow: /no/ Disallow: /se/ Disallow: /tw/ Disallow: /ua/ Disallow: /uk/ User-agent: ChatGPT-User Disallow: /au/ Disallow: /cn/ Disallow: /dk/ Disallow: /en/ Disallow: /es/ Disallow: /in/ Disallow: /it/ Disallow: /jp/ Disallow: /ng/ Disallow: /nl/ Disallow: /no/ Disallow: /se/ Disallow: /tw/ Disallow: /ua/ Disallow: /uk/ User-agent: GPTBot Disallow: /au/ Disallow: /cn/ Disallow: /dk/ Disallow: /en/ Disallow: /es/ Disallow: /in/ Disallow: /it/ Disallow: /jp/ Disallow: /ng/ Disallow: /nl/ Disallow: /no/ Disallow: /se/ Disallow: /tw/ Disallow: /ua/ Disallow: /uk/ User-agent: * Disallow: /_p/ Disallow: /afflink/ Disallow: /ajax/ Disallow: /auth/ Disallow: /au/ajax/ Disallow: /cn/ajax/ Disallow: /dk/ajax/ Disallow: /en/ajax/ Disallow: /es/ajax/ Disallow: /in/ajax/ Disallow: /it/ajax/ Disallow: /jp/ajax/ Disallow: /ng/ajax/ Disallow: /nl/ajax/ Disallow: /no/ajax/ Disallow: /se/ajax/ Disallow: /tw/ajax/ Disallow: /ua/ajax/ Disallow: /uk/ajax/ Disallow: /landing-feed/ Disallow: /oauth/ Disallow: /preview/ Disallow: /au/preview/ Disallow: /cn/preview/ Disallow: /dk/preview/ Disallow: /en/preview/ Disallow: /es/preview/ Disallow: /in/preview/ Disallow: /it/preview/ Disallow: /jp/preview/ Disallow: /ng/preview/ Disallow: /nl/preview/ Disallow: /no/preview/ Disallow: /se/preview/ Disallow: /tw/preview/ Disallow: /ua/preview/ Disallow: /uk/preview/ Disallow: /search Disallow: /transporter/ Allow: /ajax/content-product/ Allow: /au/ajax/content-product/ Allow: /cn/ajax/content-product/ Allow: /dk/ajax/content-product/ Allow: /en/ajax/content-product/ Allow: /es/ajax/content-product/ Allow: /in/ajax/content-product/ Allow: /it/ajax/content-product/ Allow: /jp/ajax/content-product/ Allow: /ng/ajax/content-product/ Allow: /nl/ajax/content-product/ Allow: /no/ajax/content-product/ Allow: /se/ajax/content-product/ Allow: /tw/ajax/content-product/ Allow: /ua/ajax/content-product/ Allow: /uk/ajax/content-product/ Allow: /ajax/contentmedia/ Allow: /au/ajax/contentmedia/ Allow: /cn/ajax/contentmedia/ Allow: /dk/ajax/contentmedia/ Allow: /en/ajax/contentmedia/ Allow: /es/ajax/contentmedia/ Allow: /in/ajax/contentmedia/ Allow: /it/ajax/contentmedia/ Allow: /jp/ajax/contentmedia/ Allow: /ng/ajax/contentmedia/ Allow: /nl/ajax/contentmedia/ Allow: /no/ajax/contentmedia/ Allow: /se/ajax/contentmedia/ Allow: /tw/ajax/contentmedia/ Allow: /ua/ajax/contentmedia/ Allow: /uk/ajax/contentmedia/ Sitemap: https://www.delish.com/sitemap_index.xml Sitemap: https://www.delish.com/sitemap_google_news.xml