Free tools

Robots.txt Examples

Robots.txt file content for flickr.com.

Robot.txt file for: flickr.com

      User-agent: Adsbot-Google
User-agent: Alexabot
User-agent: AppleBot
User-agent: archive.org_bot
User-agent: baiduspider
User-agent: bitlybot
User-agent: bingbot
User-agent: BingPreview
User-agent: coccocbot
User-agent: Discordbot
User-agent: DuckDuckBot
User-agent: facebookexternalhit
User-agent: Google-Site-Verification
User-agent: Google-Sitemaps
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: gsa-crawler
User-Agent: HatenaBlog-bot
User-agent: ia_archiver
User-agent: Mediapartners-Google
User-agent: msnbot
User-agent: NAVER
User-agent: PetalBot
User-agent: Pingdom
User-agent: Pinterest
User-agent: redditbot
User-agent: Slackbot
User-agent: Slurp
User-agent: snapchat
User-agent: Twitterbot
User-agent: wp.com
User-agent: Yandex
User-agent: YandexImages
User-agent: YandexVideoParser
User-agent: Yeti
User-agent: ZoomBot
Disallow: /gp/
Disallow: /report_abuse.gne
Disallow: /abuse
Disallow: /images/*
Disallow: /apps/*
Disallow: /tools/demos/*
Disallow: /search
Disallow: /services/oauth
Disallow: /groups/10millionphotos/
Disallow: /photos/youpy/
Disallow: /photos/i_love_u_get_away_from_me/
Disallow: /faves-i_love_u_get_away_from_me/
Disallow: /photos/gbachelie/
Disallow: /photos/archivesact/6011019532/nearby/
Disallow: /yss_fragment.gne
Disallow: /photos/tags/*/page*
Disallow: /photos/*/lightbox/
Disallow: /groups/*/lightbox/
Disallow: /explore/*/lightbox/

User-agent: magpie-crawler
Disallow: /

User-agent: *
Disallow: /

Sitemap: https://www.flickr.com/sitemap/index/users/sitemap-index-users-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/tags/sitemap-index-tags-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/sets/sitemap-index-sets-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/photos/sitemap-index-photos-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/groups/sitemap-index-groups-00000000.xml.gz