Free tools

Robots.txt Examples

Robots.txt file content for jalan.net.

Robot.txt file for: jalan.net

      User-agent: *
Sitemap: https://www.jalan.net/sitemapxml/kaigai/abroad_sitemap.xml
Sitemap: https://www.jalan.net/sitemapxml/dp/pc_dp.xml
Sitemap: https://www.jalan.net/sitemapxml/sitemap/gourmet/pc_gourmetguide.xml
Sitemap: https://www.jalan.net/sitemapxml/sitemap/travel/pc_travelguide.xml
Sitemap: https://www.jalan.net/sitemapxml/kankou/pc_kankouguide.xml
Sitemap: https://www.jalan.net/sitemapxml/sitemap/event/pc_eventguide.xml
Sitemap: https://www.jalan.net/sitemapxml/old_toksyu.xml
Sitemap: https://www.jalan.net/sitemapxml/dp/pc_dp.xml
Sitemap: https://www.jalan.net/sitemapxml/pc_static.xml
Sitemap: https://www.jalan.net/rentacar/sitemap.xml
Sitemap: https://www.jalan.net/news/author-sitemap.xml
Sitemap: https://www.jalan.net/news/sitemap_index.xml
Sitemap: https://www.jalan.net/sitemapxml/kankou/pc_kankou_spt_photo_index.xml
Sitemap: https://www.jalan.net/sitemapxml/sitemap/gourmet/pc_gourmet_grm_photo_index.xml
Sitemap: https://www.jalan.net/sitemapxml/pc_asteria.xml
Sitemap: https://www.jalan.net/sitemapxml/attentionword/pc_attention_word.xml
Disallow: /ji/service/
Disallow: /cw/
Disallow: /ct/ctp1000/
Disallow: /yw/
Disallow: /sw/
Disallow: /mt/
Disallow: /sm/
Disallow: /cl/
Disallow: /uw/uwp8100/uww8101.do
Disallow: /smart/snr/
Disallow: /us/snr/
Disallow: /dp/air/
Disallow: /pj/
Disallow: /jj/
Disallow: /campaign/secret2016/
Disallow: /campaign/secret2017/
Disallow: /campaign/secret2017_onsen/
Disallow: /campaign/secret2017_jal/
Disallow: /campaign/secret2017_service/
Disallow: /gw/
Disallow: /majibu/
Allow: /majibu/$
Allow: /majibu/concept/$