How does the appliance index a web site?

The Google.com index contains billions of webpages. The Google Mini uses this same technology to index web content via HTTP and HTTPS protocols. The administrator specifies a series of starting URLs as well as a pattern of subsequent URLs to follow. The crawler will begin by retrieving the content at the start URL and then retrieving all of the content on subsequent included URLs that correspond to the follow pattern. An example start URL is http://www.mycompany.com. The Google Mini will stop indexing content once it reaches its document count license limit or when there is no more content to retrieve.