Free tools
Robots.txt Examples
Robots.txt file content for mapcarta.com.
Robot.txt file for: mapcarta.com
# based on "The Ultimate robots.txt Bot and User-Agent Blocker" # copyright, MIT License: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker User-agent: 01h4x.com User-agent: 360Spider User-agent: 404checker User-agent: 404enemy User-agent: 80legs User-agent: ADmantX User-agent: AIBOT User-agent: ALittle Client User-agent: ASPSeek User-agent: Abonti User-agent: Aboundex User-agent: Aboundexbot User-agent: Acunetix User-agent: AdsTxtCrawlerTP User-agent: AfD-Verbotsverfahren User-agent: AhrefsBot User-agent: AiHitBot User-agent: Aipbot User-agent: Alexibot User-agent: AllSubmitter User-agent: Alligator User-agent: AlphaBot User-agent: Anarchie User-agent: Anarchy User-agent: Anarchy99 User-agent: Ankit User-agent: Anthill User-agent: Apexoo User-agent: Aspiegel User-agent: Asterias User-agent: Atomseobot User-agent: Attach User-agent: AwarioBot User-agent: AwarioRssBot User-agent: AwarioSmartBot User-agent: BBBike User-agent: BDCbot User-agent: BDFetch User-agent: BLEXBot User-agent: BackDoorBot User-agent: BackStreet User-agent: BackWeb User-agent: Backlink-Ceck User-agent: BacklinkCrawler User-agent: BacklinksExtendedBot User-agent: Badass User-agent: Bandit User-agent: Barkrowler User-agent: BatchFTP User-agent: Battleztar Bazinga User-agent: BetaBot User-agent: Bigfoot User-agent: Bitacle User-agent: BlackWidow User-agent: Black Hole User-agent: Blackboard User-agent: Blow User-agent: BlowFish User-agent: Boardreader User-agent: Bolt User-agent: BotALot User-agent: Brandprotect User-agent: Brandwatch User-agent: Buck User-agent: Buddy User-agent: BuiltBotTough User-agent: BuiltWith User-agent: Bullseye User-agent: BunnySlippers User-agent: BuzzSumo User-agent: Bytespider User-agent: CATExplorador User-agent: CCBot User-agent: CODE87 User-agent: CSHttp User-agent: Calculon User-agent: CazoodleBot User-agent: Cegbfeieh User-agent: CensysInspect User-agent: CheTeam User-agent: CheeseBot User-agent: CherryPicker User-agent: ChinaClaw User-agent: Chlooe User-agent: Citoid User-agent: Claritybot User-agent: Cliqzbot User-agent: Cloud mapping User-agent: Cocolyzebot User-agent: Cogentbot User-agent: Collector User-agent: Copier User-agent: CopyRightCheck User-agent: Copyscape User-agent: Cosmos User-agent: Craftbot User-agent: Crawling at Home Project User-agent: CrazyWebCrawler User-agent: Crescent User-agent: CrunchBot User-agent: Curious User-agent: Custo User-agent: CyotekWebCopy User-agent: DBLBot User-agent: DIIbot User-agent: DSearch User-agent: DTS Agent User-agent: DataCha0s User-agent: DatabaseDriverMysqli User-agent: Demon User-agent: Deusu User-agent: Devil User-agent: Digincore User-agent: DigitalPebble User-agent: Dirbuster User-agent: Disco User-agent: Discobot User-agent: Discoverybot User-agent: Dispatch User-agent: DittoSpyder User-agent: DnBCrawler-Analytics User-agent: DnyzBot User-agent: DomCopBot User-agent: DomainAppender User-agent: DomainCrawler User-agent: DomainSigmaCrawler User-agent: DomainStatsBot User-agent: Domains Project User-agent: Dotbot User-agent: Download Wonder User-agent: Dragonfly User-agent: Drip User-agent: ECCP/1.0 User-agent: EMail Siphon User-agent: EMail Wolf User-agent: EasyDL User-agent: Ebingbong User-agent: Ecxi User-agent: EirGrabber User-agent: EroCrawler User-agent: Evil User-agent: Exabot User-agent: Express WebPictures User-agent: ExtLinksBot User-agent: Extractor User-agent: ExtractorPro User-agent: Extreme Picture Finder User-agent: EyeNetIE User-agent: Ezooms User-agent: FDM User-agent: FHscan User-agent: FemtosearchBot User-agent: Fimap User-agent: Firefox/7.0 User-agent: FlashGet User-agent: Flunky User-agent: Foobot User-agent: Freeuploader User-agent: FrontPage User-agent: Fuzz User-agent: FyberSpider User-agent: Fyrebot User-agent: G-i-g-a-b-o-t User-agent: GT::WWW User-agent: GalaxyBot User-agent: Genieo User-agent: GermCrawler User-agent: GetRight User-agent: GetWeb User-agent: Getintent User-agent: Gigabot User-agent: Go!Zilla User-agent: Go-Ahead-Got-It User-agent: GoZilla User-agent: Gotit User-agent: GrabNet User-agent: Grabber User-agent: Grafula User-agent: GrapeFX User-agent: GrapeshotCrawler User-agent: GridBot User-agent: HEADMasterSEO User-agent: HMView User-agent: HTMLparser User-agent: HTTP::Lite User-agent: HTTrack User-agent: Haansoft User-agent: HaosouSpider User-agent: Harvest User-agent: Havij User-agent: Heritrix User-agent: Hloader User-agent: HonoluluBot User-agent: Humanlinks User-agent: HybridBot User-agent: IDBTE4M User-agent: IDBot User-agent: IRLbot User-agent: Iblog User-agent: Id-search User-agent: IlseBot User-agent: Image Fetch User-agent: Image Sucker User-agent: IndeedBot User-agent: Indy Library User-agent: InfoNaviRobot User-agent: InfoTekies User-agent: Information Security Team InfraSec Scanner User-agent: InfraSec Scanner User-agent: Intelliseek User-agent: InterGET User-agent: InternetMeasurement User-agent: InternetSeer User-agent: Internet Ninja User-agent: Iria User-agent: Iskanie User-agent: IstellaBot User-agent: JOC Web Spider User-agent: JamesBOT User-agent: Jbrofuzz User-agent: JennyBot User-agent: JetCar User-agent: Jetty User-agent: JikeSpider User-agent: Joomla User-agent: Jorgee User-agent: JustView User-agent: Jyxobot User-agent: Kenjin Spider User-agent: Keybot Translation-Search-Machine User-agent: Keyword Density User-agent: Kinza User-agent: Kozmosbot User-agent: LNSpiderguy User-agent: LWP::Simple User-agent: Lanshanbot User-agent: Larbin User-agent: Leap User-agent: LeechFTP User-agent: LeechGet User-agent: LexiBot User-agent: Lftp User-agent: LibWeb User-agent: Libwhisker User-agent: LieBaoFast User-agent: Lightspeedsystems User-agent: Likse User-agent: LinkScan User-agent: LinkWalker User-agent: Linkbot User-agent: LinkextractorPro User-agent: LinkpadBot User-agent: LinksManager User-agent: LinqiaMetadataDownloaderBot User-agent: LinqiaRSSBot User-agent: LinqiaScrapeBot User-agent: Lipperhey User-agent: Lipperhey Spider User-agent: Litemage_walker User-agent: Lmspider User-agent: Ltx71 User-agent: MFC_Tear_Sample User-agent: MIDown tool User-agent: MIIxpc User-agent: MJ12bot User-agent: MQQBrowser User-agent: MSFrontPage User-agent: MSIECrawler User-agent: MTRobot User-agent: Mag-Net User-agent: Magnet User-agent: Mail.RU_Bot User-agent: Majestic-SEO User-agent: Majestic12 User-agent: Majestic SEO User-agent: MarkMonitor User-agent: MarkWatch User-agent: Mass Downloader User-agent: Masscan User-agent: Mata Hari User-agent: MauiBot User-agent: Mb2345Browser User-agent: MeanPath Bot User-agent: Meanpathbot User-agent: Mediatoolkitbot User-agent: MegaIndex.ru User-agent: Metauri User-agent: MicroMessenger User-agent: Microsoft Data Access User-agent: Microsoft URL Control User-agent: Minefield User-agent: Mister PiX User-agent: Moblie Safari User-agent: Mojeek User-agent: Mojolicious User-agent: MolokaiBot User-agent: Morfeus Fucking Scanner User-agent: Mozlila User-agent: Mr.4x3 User-agent: Msrabot User-agent: Musobot User-agent: NICErsPRO User-agent: NPbot User-agent: Name Intelligence User-agent: Nameprotect User-agent: Navroad User-agent: NearSite User-agent: Needle User-agent: Nessus User-agent: NetAnts User-agent: NetLyzer User-agent: NetMechanic User-agent: NetSpider User-agent: NetZIP User-agent: Net Vampire User-agent: Netcraft User-agent: Nettrack User-agent: Netvibes User-agent: NextGenSearchBot User-agent: Nibbler User-agent: Niki-bot User-agent: Nikto User-agent: NimbleCrawler User-agent: Nimbostratus User-agent: Ninja User-agent: Nmap User-agent: Nuclei User-agent: Nutch User-agent: Octopus User-agent: Offline Explorer User-agent: Offline Navigator User-agent: OnCrawl User-agent: OpenLinkProfiler User-agent: OpenVAS User-agent: Openfind User-agent: Openvas User-agent: OrangeBot User-agent: OrangeSpider User-agent: OutclicksBot User-agent: OutfoxBot User-agent: PECL::HTTP User-agent: PHPCrawl User-agent: POE-Component-Client-HTTP User-agent: PageAnalyzer User-agent: PageGrabber User-agent: PageScorer User-agent: PageThing.com User-agent: Page Analyzer User-agent: Pandalytics User-agent: Panscient User-agent: Papa Foto User-agent: Pavuk User-agent: PeoplePal User-agent: Petalbot User-agent: Pi-Monster User-agent: Picscout User-agent: Picsearch User-agent: PictureFinder User-agent: Piepmatz User-agent: Pimonster User-agent: Pixray User-agent: PleaseCrawl User-agent: Pockey User-agent: ProPowerBot User-agent: ProWebWalker User-agent: Probethenet User-agent: Proximic User-agent: Psbot User-agent: Pu_iN User-agent: Pump User-agent: PxBroker User-agent: PyCurl User-agent: QueryN Metasearch User-agent: Quick-Crawler User-agent: RSSingBot User-agent: Rainbot User-agent: RankActive User-agent: RankActiveLinkBot User-agent: RankFlex User-agent: RankingBot User-agent: RankingBot2 User-agent: Rankivabot User-agent: RankurBot User-agent: Re-re User-agent: ReGet User-agent: RealDownload User-agent: Reaper User-agent: RebelMouse User-agent: Recorder User-agent: RedesScrapy User-agent: RepoMonkey User-agent: Ripper User-agent: RocketCrawler User-agent: Rogerbot User-agent: SBIder User-agent: SEOkicks User-agent: SEOkicks-Robot User-agent: SEOlyticsCrawler User-agent: SEOprofiler User-agent: SEOstats User-agent: SISTRIX User-agent: SMTBot User-agent: SalesIntelligent User-agent: ScanAlert User-agent: Scanbot User-agent: ScoutJet User-agent: Scrapy User-agent: Screaming User-agent: ScreenerBot User-agent: ScrepyBot User-agent: Searchestate User-agent: SearchmetricsBot User-agent: Seekport User-agent: SeekportBot User-agent: SemanticJuice User-agent: Semrush User-agent: SemrushBot User-agent: SentiBot User-agent: SenutoBot User-agent: SEOlyt User-agent: SeoSiteCheckup User-agent: SeobilityBot User-agent: Seomoz User-agent: Shodan User-agent: Siphon User-agent: SiteCheckerBotCrawler User-agent: SiteExplorer User-agent: SiteLockSpider User-agent: SiteSnagger User-agent: SiteSucker User-agent: Site Sucker User-agent: Sitebeam User-agent: Siteimprove User-agent: Sitevigil User-agent: SlySearch User-agent: SmartDownload User-agent: Snake User-agent: Snapbot User-agent: Snoopy User-agent: SocialRankIOBot User-agent: Sociscraper User-agent: Sogou web spider User-agent: Sosospider User-agent: Sottopop User-agent: SpaceBison User-agent: Spammen User-agent: SpankBot User-agent: Spanner User-agent: Spbot User-agent: Spider_Bot User-agent: Spider_Bot/3.0 User-agent: Spinn3r User-agent: SputnikBot User-agent: Sqlmap User-agent: Sqlworm User-agent: Sqworm User-agent: Steeler User-agent: Stripper User-agent: Sucker User-agent: Sucuri User-agent: SuperBot User-agent: SuperHTTP User-agent: Surfbot User-agent: SurveyBot User-agent: Suzuran User-agent: Swiftbot User-agent: Szukacz User-agent: T0PHackTeam User-agent: T8Abot User-agent: Teleport User-agent: TeleportPro User-agent: Telesoft User-agent: Telesphoreo User-agent: Telesphorep User-agent: TheNomad User-agent: The Intraformant User-agent: Thumbor User-agent: TightTwatBot User-agent: TinyTestBot User-agent: Titan User-agent: Toata User-agent: Toweyabot User-agent: Tracemyfile User-agent: Trendiction User-agent: Trendictionbot User-agent: True_Robot User-agent: Turingos User-agent: Turnitin User-agent: TurnitinBot User-agent: TwengaBot User-agent: Twice User-agent: Typhoeus User-agent: URLy.Warning User-agent: URLy Warning User-agent: UnisterBot User-agent: Upflow User-agent: V-BOT User-agent: VB Project User-agent: VCI User-agent: Vacuum User-agent: Vagabondo User-agent: VelenPublicWebCrawler User-agent: VeriCiteCrawler User-agent: VidibleScraper User-agent: Virusdie User-agent: VoidEYE User-agent: Voil User-agent: Voltron User-agent: WASALive-Bot User-agent: WBSearchBot User-agent: WEBDAV User-agent: WISENutbot User-agent: WPScan User-agent: WWW-Collector-E User-agent: WWW-Mechanize User-agent: WWW::Mechanize User-agent: WWWOFFLE User-agent: Wallpapers User-agent: Wallpapers/3.0 User-agent: WallpapersHD User-agent: WeSEE User-agent: WebAuto User-agent: WebBandit User-agent: WebCollage User-agent: WebCopier User-agent: WebEnhancer User-agent: WebFetch User-agent: WebFuck User-agent: WebGo IS User-agent: WebImageCollector User-agent: WebLeacher User-agent: WebPix User-agent: WebReaper User-agent: WebSauger User-agent: WebStripper User-agent: WebSucker User-agent: WebWhacker User-agent: WebZIP User-agent: Web Auto User-agent: Web Collage User-agent: Web Enhancer User-agent: Web Fetch User-agent: Web Fuck User-agent: Web Pix User-agent: Web Sauger User-agent: Web Sucker User-agent: Webalta User-agent: WebmasterWorldForumBot User-agent: Webshag User-agent: WebsiteExtractor User-agent: WebsiteQuester User-agent: Website Quester User-agent: Webster User-agent: Whack User-agent: Whacker User-agent: Whatweb User-agent: Who.is Bot User-agent: Widow User-agent: WinHTTrack User-agent: WiseGuys Robot User-agent: Wonderbot User-agent: Woobot User-agent: Wotbox User-agent: Wprecon User-agent: Xaldon_WebSpider User-agent: Xenu User-agent: YaK User-agent: YoudaoBot User-agent: Zade User-agent: Zauba User-agent: Zermelo User-agent: Zeus User-agent: Zitebot User-agent: ZmEu User-agent: ZoomBot User-agent: ZoominfoBot User-agent: ZumBot User-agent: ZyBorg User-agent: adscanner User-agent: archive.org_bot User-agent: arquivo-web-crawler User-agent: arquivo.pt User-agent: autoemailspider User-agent: awario.com User-agent: backlink-check User-agent: cah.io.community User-agent: check1.exe User-agent: clark-crawler User-agent: coccocbot User-agent: cognitiveseo User-agent: com.plumanalytics User-agent: crawl.sogou.com User-agent: crawler.feedback User-agent: crawler4j User-agent: dataforseo.com User-agent: dataforseobot User-agent: demandbase-bot User-agent: domainsproject.org User-agent: eCatch User-agent: evc-batch User-agent: facebookscraper User-agent: gopher User-agent: heritrix User-agent: instabid User-agent: internetVista monitor User-agent: ips-agent User-agent: isitwp.com User-agent: iubenda-radar User-agent: linkdexbot User-agent: linkfluence User-agent: lwp-request User-agent: lwp-trivial User-agent: magpie-crawler User-agent: meanpathbot User-agent: mediawords User-agent: muhstik-scan User-agent: netEstate NE Crawler User-agent: oBot User-agent: omgili User-agent: page scorer User-agent: pcBrowser User-agent: plumanalytics User-agent: polaris version User-agent: probe-image-size User-agent: ripz User-agent: s1z.ru User-agent: satoristudio.net User-agent: scalaj-http User-agent: scan.lol User-agent: seobility User-agent: seocompany.store User-agent: seoscanners User-agent: seostar User-agent: serpstatbot User-agent: sexsearcher User-agent: sitechecker.pro User-agent: siteripz User-agent: sogouspider User-agent: sp_auditbot User-agent: spyfu User-agent: sysscan User-agent: tAkeOut User-agent: trendiction.com User-agent: trendiction.de User-agent: ubermetrics-technologies.com User-agent: voyagerx.com User-agent: webgains-bot User-agent: webmeup-crawler User-agent: webpros.com User-agent: webprosbot User-agent: x09Mozilla User-agent: x22Mozilla User-agent: xpymep1.exe User-agent: zauba.io User-agent: zgrab User-agent: Bytedance User-agent: Morningscore User-agent: Pumoxbot User-agent: SurdotlyBot User-agent: WRTNBot User-agent: ev-crawler Disallow: / User-agent: * Allow: / Disallow: /s/ Disallow: /de/s/ Disallow: /es/s/ Disallow: /fr/s/ Disallow: /pt/s/ Disallow: /dynamic/routing.php Disallow: /dynamic/search.php Sitemap: https://mapcarta.com/sitemap.xml