Free tools

Robots.txt Examples

Robots.txt file content for timeout.com.

Robot.txt file for: timeout.com

      # robots.txt file for Time Out .com
# updated 14th May 2020

User-agent: *
Disallow: /graffiti/v1/
Disallow: /user/ajax/
Disallow: */pages-ajax
Disallow: */feed.rss
Disallow: */paginate
Disallow: /user/mtoPopupLogin
Disallow: /ajax/mastercard-theme
Disallow: */user/sign-up
Disallow: *user/myAccount
Disallow: *user/forgotPassword
Disallow: */user/email/review
Disallow: */user/token
Disallow: /user//attending/upcoming*
Disallow: */rss
Disallow: */rss_en_all
Disallow: */rss_ja_all
Disallow: */rss/Features
Disallow: */rss/Events
Disallow: */rss_for_twitter
Disallow: */rss/Venues
Disallow: /search*
Disallow: /*/search*
Disallow: /*/search/
Disallow: /*/search
Disallow: /*/*/search*
Disallow: /*/*/search/
Disallow: /*/*/search
Disallow: /search-listings?*
Disallow: /*/*.php
Disallow: /*/*/bookings

Sitemap: https://www.timeout.com/sitemap.xml.gz
Sitemap: https://www.timeout.com/bangkok/news_sitemap.xml
Sitemap: https://www.timeout.com/bangkok/th/news_sitemap.xml
Sitemap: https://www.timeout.com/chicago/news_sitemap.xml
Sitemap: https://www.timeout.com/croatia/news_sitemap.xml
Sitemap: https://www.timeout.com/hong-kong/news_sitemap.xml
Sitemap: https://www.timeout.com/israel/news_sitemap.xml
Sitemap: https://www.timeout.com/kuala-lumpur/news_sitemap.xml
Sitemap: https://www.timeout.com/london/news_sitemap.xml
Sitemap: https://www.timeout.com/los-angeles/news_sitemap.xml
Sitemap: https://www.timeout.com/melbourne/news_sitemap.xml
Sitemap: https://www.timeout.com/miami/news_sitemap.xml
Sitemap: https://www.timeout.com/newyork/news_sitemap.xml
Sitemap: https://www.timeout.com/new-york-kids/news_sitemap.xml
Sitemap: https://www.timeout.com/san-francisco/news_sitemap.xml
Sitemap: https://www.timeout.com/singapore/news_sitemap.xml
Sitemap: https://www.timeout.com/sri-lanka/news_sitemap.xml
Sitemap: https://www.timeout.com/sydney/news_sitemap.xml
Sitemap: https://www.timeout.com/tokyo/news_sitemap.xml
Sitemap: https://www.timeout.com/usa/news_sitemap.xml

User-agent: 80bot
Disallow: /

User-agent: BecomeBot
Crawl-delay: 15

User-agent: genieBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: spock
Disallow: /

User-agent: TurnitinBot
Crawl-delay: 15