Free tools
Robots.txt Examples
Robots.txt file content for udn.com.
Robot.txt file for: udn.com
# robots.txt User-agent: * #Disallow: /.well-known/amphtml/apikey.pub Disallow: /BT/* Disallow: /fcm/* Disallow: /NEWS/* Disallow: /NEWS/BREAKINGNEWS/* Disallow: /Project/* Disallow: /UDN/FOUNDER/* Disallow: /UDN/UDNENGLISH/* Disallow: /morakot/* Allow: /page/topic/184 Allow: /page/topic/495 Allow: /page/topic/496 Disallow: /page/topic/* Sitemap: https://udn.com/sitemapxml/news/mapindex.xml Sitemap: https://udn.com/sitemap/gnews/2 Sitemap: https://udn.com/sitemap/gnews/1013 Sitemap: https://udn.com/sitemap/gnews/1015 # chat bot User-agent: GPTBot Disallow: / User-agent: Applebot Disallow: / User-agent: Amazonbot Disallow: / # another bot User-agent: trendictionbot Disallow: /