Free tools

Robots.txt Examples

Robots.txt file content for udn.com.

Robot.txt file for: udn.com

      # KB13270 robots.txt
User-agent: *
Disallow: /srank/*
#Disallow: /.well-known/amphtml/apikey.pub
Disallow: /BT/*
Disallow: /fcm/*
Disallow: /NEWS/*
Disallow: /NEWS/BREAKINGNEWS/*
Disallow: /Project/*
Disallow: /UDN/FOUNDER/*
Disallow: /UDN/UDNENGLISH/*
Disallow: /morakot/*
Allow: /page/topic/184
Allow: /page/topic/495
Allow: /page/topic/496 
Disallow: /page/topic/*
Sitemap: https://udn.com/sitemapxml/news/mapindex.xml
Sitemap: https://udn.com/sitemap/gnews/2
Sitemap: https://udn.com/sitemap/gnews/1013
Sitemap: https://udn.com/sitemap/gnews/1015
User-agent: GPTBot
Disallow: /
User-agent: Claude
Disallow: /
User-agent: ClaudeBot
Disallow: /