Does the Mini honor robots.txt and robots meta tags?
The Google Mini will honor robots.txt files and robot meta tags in the documents that it crawls. Robots.txt is a plain text file that you create and put on your server to exclude search engine crawlers from accessing pages or directories on your site. Robots meta tags allow you to embed a meta tag element on pages or documents you do not want crawled. This helps the Mini integrate into an existing web site, and gives you another way to control how the Mini crawls your documents.