Free tools

Robots.txt Examples

Robots.txt file content for smule.com.

Robot.txt file for: smule.com

      User-agent: PiplBot
Disallow: /

User-agent: piplbot
Disallow: /

User-agent: EasouSpider
Disallow: /

User-agent: meanpathbot
Disallow: /

User-agent: QuerySeekerSpider
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: *
Disallow: /s/js-log
Disallow: /user/
Disallow: /password_reset
Disallow: /activate
Disallow: /s/playlist_json
Disallow: /profile/block
Disallow: /profile/unblock
Disallow: /profile/is_blocked
Disallow: /forum/
Disallow: /s/performance/

Sitemap: https://www.smule.com/assets/sitemaps/daily.xml.gz
Sitemap: https://www.smule.com/assets/sitemaps/weekly.xml.gz
Sitemap: https://www.smule.com/assets/sitemaps/monthly.xml.gz
Sitemap: https://www.smule.com/assets/sitemaps/yearly.xml.gz