Free tools
Robots.txt Examples
Robots.txt file content for flickr.com.
Robot.txt file for: flickr.com
User-agent: Adsbot-Google User-agent: Alexabot User-agent: AppleBot User-agent: archive.org_bot User-agent: baiduspider User-agent: bitlybot User-agent: bingbot User-agent: BingPreview User-agent: coccocbot User-agent: Discordbot User-agent: DuckDuckBot User-agent: facebookexternalhit User-agent: Google-Site-Verification User-agent: Google-Sitemaps User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-Mobile User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: gsa-crawler User-Agent: HatenaBlog-bot User-agent: ia_archiver User-agent: Mediapartners-Google User-agent: msnbot User-agent: NAVER User-agent: PetalBot User-agent: Pingdom User-agent: Pinterest User-agent: redditbot User-agent: Slackbot User-agent: Slurp User-agent: snapchat User-agent: Twitterbot User-agent: wp.com User-agent: Yandex User-agent: YandexImages User-agent: YandexVideoParser User-agent: Yeti User-agent: ZoomBot Disallow: /gp/ Disallow: /report_abuse.gne Disallow: /abuse Disallow: /images/* Disallow: /apps/* Disallow: /tools/demos/* Disallow: /search Disallow: /services/oauth Disallow: /groups/10millionphotos/ Disallow: /photos/youpy/ Disallow: /photos/i_love_u_get_away_from_me/ Disallow: /faves-i_love_u_get_away_from_me/ Disallow: /photos/gbachelie/ Disallow: /photos/archivesact/6011019532/nearby/ Disallow: /yss_fragment.gne Disallow: /photos/tags/*/page* Disallow: /photos/*/lightbox/ Disallow: /groups/*/lightbox/ Disallow: /explore/*/lightbox/ User-agent: magpie-crawler Disallow: / User-agent: * Disallow: / Sitemap: https://www.flickr.com/sitemap/index/users/sitemap-index-users-00000000.xml.gz Sitemap: https://www.flickr.com/sitemap/index/tags/sitemap-index-tags-00000000.xml.gz Sitemap: https://www.flickr.com/sitemap/index/sets/sitemap-index-sets-00000000.xml.gz Sitemap: https://www.flickr.com/sitemap/index/photos/sitemap-index-photos-00000000.xml.gz Sitemap: https://www.flickr.com/sitemap/index/groups/sitemap-index-groups-00000000.xml.gz