Free tools

Robots.txt Examples

Robots.txt file content for udn.com.

Robot.txt file for: udn.com

      # robots.txt
User-agent: *
#Disallow: /.well-known/amphtml/apikey.pub
Disallow: /BT/*
Disallow: /fcm/*
Disallow: /NEWS/*
Disallow: /NEWS/BREAKINGNEWS/*
Disallow: /Project/*
Disallow: /UDN/FOUNDER/*
Disallow: /UDN/UDNENGLISH/*
Disallow: /morakot/*
Allow: /page/topic/184
Allow: /page/topic/495
Allow: /page/topic/496 
Disallow: /page/topic/*
Sitemap: https://udn.com/sitemapxml/news/mapindex.xml
Sitemap: https://udn.com/sitemap/gnews/2
Sitemap: https://udn.com/sitemap/gnews/1013
Sitemap: https://udn.com/sitemap/gnews/1015
# chat bot
User-agent: GPTBot
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: Amazonbot
Disallow: /
# another bot
User-agent: trendictionbot
Disallow: /