Free tools
Robots.txt Examples
Robots.txt file content for scmp.com.
Robot.txt file for: scmp.com
User-agent: * crawl-delay: 10 # Sitemap Sitemap: https://www.scmp.com/sitemap.xml Sitemap: https://www.scmp.com/sitemap_explained.xml Sitemap: https://www.scmp.com/sitemap_podcasts.xml Sitemap: https://www.scmp.com/sitemap_announcements.xml Sitemap: https://www.scmp.com/sitemap_infographics.xml Sitemap: https://www.scmp.com/sitemap_news.xml Sitemap: https://www.scmp.com/sitemap_economy.xml Sitemap: https://www.scmp.com/sitemap_business.xml Sitemap: https://www.scmp.com/sitemap_comment.xml Sitemap: https://www.scmp.com/sitemap_tech.xml Sitemap: https://www.scmp.com/sitemap_lifestyle.xml Sitemap: https://www.scmp.com/sitemap_culture.xml Sitemap: https://www.scmp.com/sitemap_sport.xml Sitemap: https://www.scmp.com/sitemap_property.xml Sitemap: https://www.scmp.com/sitemap_photos.xml Sitemap: https://www.scmp.com/sitemap_video.xml Sitemap: https://www.scmp.com/sitemap_destination_macau.xml Sitemap: https://www.scmp.com/sitemap_magazines.xml Sitemap: https://www.scmp.com/sitemap_this_week_in_asia.xml Sitemap: https://www.scmp.com/sitemap_directories.xml Sitemap: https://www.scmp.com/sitemap_weather.xml Sitemap: https://www.scmp.com/sitemap_about_us.xml Sitemap: https://www.scmp.com/sitemap_lists.xml Sitemap: https://www.scmp.com/sitemap_special_reports.xml Sitemap: https://www.scmp.com/sitemap_country_reports.xml Sitemap: https://www.scmp.com/sitemap_video_comments.xml Sitemap: https://www.scmp.com/sitemap_video_scmp_originals.xml Sitemap: https://www.scmp.com/sitemap_video_hong_kong.xml Sitemap: https://www.scmp.com/sitemap_video_china.xml Sitemap: https://www.scmp.com/sitemap_video_asia.xml Sitemap: https://www.scmp.com/sitemap_video_world.xml Sitemap: https://www.scmp.com/sitemap_video_business.xml Sitemap: https://www.scmp.com/sitemap_video_arts_culture.xml Sitemap: https://www.scmp.com/sitemap_video_technology.xml Sitemap: https://www.scmp.com/sitemap_video_lifestyle.xml Sitemap: https://www.scmp.com/sitemap_video_sport.xml Sitemap: https://www.scmp.com/sitemap_video_offbeat.xml Sitemap: https://www.scmp.com/sitemap_video_style.xml Sitemap: https://www.scmp.com/sitemap_video_post_mag.xml Sitemap: https://www.scmp.com/sitemap_video_presented.xml Sitemap: https://www.scmp.com/sitemap_article.xml Sitemap: https://www.scmp.com/sitemap_gallery.xml Sitemap: https://www.scmp.com/sitemap_poll.xml Sitemap: https://www.scmp.com/sitemap_promotion.xml Sitemap: https://www.scmp.com/sitemap_webform.xml Sitemap: https://www.scmp.com/sitemap_video_format.xml Sitemap: https://www.scmp.com/sitemap_sections.xml Sitemap: https://www.scmp.com/sitemap_topics.xml Sitemap: https://www.scmp.com/sitemap_authors.xml Sitemap: https://www.scmp.com/sitemap_archive.xml Sitemap: https://www.scmp.com/sitemap_abacus.xml Sitemap: https://www.scmp.com/sitemap_recipe.xml Sitemap: https://www.scmp.com/sitemap_better_life.xml Sitemap: https://www.scmp.com/sitemap_podcast.xml