User-agent: ia_archiver Disallow: /folder-name-not-be-archived-and-crawled-by-bots/ User-agent: Libwww-perl Disallow: /