Search
Clear search
Close search
Google apps
Main menu
true

Configuring the Connector for Lotus Notes

Connector software version 3.0
Installer version 3.0



Introducing Google Search Appliance Connector for Lotus Notes

The Google Search Appliance Connector for Lotus Notes 3.0 provides advanced search methods for IBM Lotus Notes applications that seamlessly integrate your Lotus Notes content with other enterprise content.

This document contains the information you need to the connector and to configure the Google Search Appliance (GSA) and the connector to traverse, index, and search content in Lotus Notes.

The software was designed by Lotus Notes developers and systems administrators who have an in-depth understanding of the design structure, principles, and advantages of the Notes database environment. Because of this, the system has unique features that maximize the Lotus Notes database design. For example:

  • The Notes connector enforces template definitions for each database. This allows you to fine-tune data that is searched to the level of forms and fields, and to ensure that only meaningful data is indexed.
  • Authorization reflects the multiple levels provided in Lotus Notes. The connector provides query-time authentication and authorization using the available native security mechanisms.
  • Formulas use Notes’ database structure to generate default document titles and descriptions.

To set up the Lotus Notes connector, you will need to be familiar with Lotus Notes application design to configure the connector, and you will need to work with a Lotus Notes administrator to determine the correct values for installing and configuring the connector. You will also need to work closely with the Lotus Notes administrator so that the crawler has the access level it needs.

After you install the connector, you can configure it to traverse documents and metadata in Lotus Notes applications and feed them to the search appliance for indexing. The initial traversal will feed all documents in the database that meet the selection criteria. After the initial traversal, the connector works in an incremental mode to feed documents that are added or modified.


What’s New for 3.0

The architecture has been transformed. Prior to version 2.8, a Lotus Notes application stored indexes, documents, and access information; now these are stored in an alternative database, which is faster.

  • The connector supports using wildcard certificate hierarchies in database ACLs and in security groups.
  • Deleted documents will be removed from the search index.
  • The Indexed, People, and Groups tabs in the GSA Configuration database have been removed since this information is no longer kept in the Notes connector database and has been moved to an alternative database.

Features of the New Architecture

Version 2.8 and up is a major upgrade. The following are new or changed features of the connector:

  • The connector is now implemented using the Connector Manager and runs on Apache Tomcat using the Lotus Notes client to access applications on the Domino server.
  • The Agent Manager and HTTP Server task are no longer used to run agents and are no longer required.
  • Connector authentication and authorization replace SAML authentication and authorization.
  • Connector code runs entirely in Tomcat; however the configuration database remains in Domino.
  • Significant improvements in reliability and scalability for traversal.
  • Custom metadata fields can be configured in the application template.
  • Domino application ACLs can be sent to the GSA as Policy ACLs, improving authorization performance and eliminating the need to re-index documents when application ACLs change.
  • Connector authorization enforces “Deny Access” for users in an application ACL.
  • Mapping of file extensions to MIME types is exposed.
  • Attachments are extracted to a directory on the local Notes client, greatly reducing the size of the GSA connector database.
  • Up to 2MB of text in rich text fields are sent to the GSA for indexing.
  • The directory monitor now does a progressive update so that authentication and authorization can continue uninterrupted.

There is no migration from previous versions of the connector to version 2.8 or 3.0 because the connectors have completely different architectures. Components in the Google Search Appliance Connector for Lotus Notes.


Using Wildcards with ACLs

Wildcards are supported for both Document ACLs and connector authorization.

The Lotus Notes Connector crawls for wildcard configuration in the following sequences:

  1. It crawls the Lotus Notes directory to build the domain cache. Each wildcard entry is maintained as a group in an alternative database.
  2. When it crawls groups, all nested groups and wildcard members are maintained as child groups.
  3. When it crawls people, it expands the user’s domain hierarchy, looks up the user’s groups, and builds the user and groups mapping in an alternative database.
  4. When it crawls database ACLs, if the database is configured using “Document ACL”, it sends permitted users and groups in the feed. During serving time and authentication, the connector computes user’s groups and child groups and returns them for to the GSA for each doc id. If “Connector” authorization is configured, the connector authorization mechanism retrieves user’s groups from the GSA Configuration database.

Migrating from an Earlier Version

To migrate from an earlier version:

  1. Save the previous version of the configuration database in a backup file and change the title so that it reflects that it is an older version.
  2. Follow the instructions in Creating the Connector Database to create a new instance of the GSA Configuration database.
  3. Copy and paste relevant configurations from the previous database to the new one, including template definitions.

Requirements

The following components are required for the connector:

  • A Google Search Appliance.
  • A Lotus Domino domain with one or more Domino servers.
  • One or more Domino servers which will host the Google Search Appliance Connector for Lotus Notes database and Domino applications, which will be indexed.
  • The connector host with the Lotus Notes client, GSA Connector installation and Java Development Kit (JDK).

    Note: The GSA Connector installation includes Apache Tomcat, the connector manager and the Lotus Notes connector for the Google Search Appliance.
Important: For optimal performance, install the Domino server, the GSA Connector Database, the Lotus Notes client and Lotus Notes Connector on the same physical host. This is a configuration supported by IBM for running API programs.

Supported Operating Systems

 
Windows The connectors can be installed on x86/x64 Windows platforms supported by IBM for running the IBM Lotus Notes client. Refer to IBM system requirements for specific details. However, Windows Vista, Window 7, and Windows Server 2008 should be supported platforms.
Linux Linux is not a supported platform. The connector can be installed on 32 bit Linux Intel x86 that support the IBM Lotus Notes client however the connector is experimental and no support is provided for Linux platforms.
Apple Macintosh OS Mac OS is not a supported platform.

Supported Lotus Notes Version

The Google Search Appliance Connector for Lotus Notes supports the Lotus Notes client version 8.0.x and 8.5.x on Windows. The Notes Administrator is required for installing the connector.

WARNING: The Connector for Lotus Notes runs as a service on Windows. You must not have the Notes client started or start the Notes clients when the connector is running, otherwise resource sharing conflicts in the Notes API will cause the connector to crash.

Supported Domino Server Versions

The Google Search Appliance Connector for Lotus Notes supports Domino server versions running 7.0.x, 8.0.x and 8.5.x on Windows, Linux/Intel, AIX, and Solaris.


Supported Java Version

The Google Search Appliance Connector for Lotus Notes requires the 32-bit Oracle Java Development Kit version 6.


Hardware Recommendations

In addition to the IBM requirements for the Notes client (and Domino Server if they are running on the same machine) the following is recommended:

  • 3 gigabytes (GB) Memory
  • 100 - 500 GB Free Disk Space in the Notes client data directory
  • 4 GB of disk space on the Domino server
  • 2 - 4 GB of disk space on the connector manager machine

Performance Considerations: Best Practices

Consider the following best practices when deploying the Lotus Notes Connector for the Google Search Appliance:

  • For optimal crawl performance, Lotus Notes applications should be located on a Domino server in the same Local Area Network.
  • The connector should be installed on the same host as the Domino server hosting the connector database. If this is not possible, then the connector should be installed on the same subnet/network segment to minimize network latency. This is especially important when connector authorization is used.
  • If you do not require security for the documents being indexed, select No security for Document Authorization when registering Lotus Notes applications.
  • If your Lotus Notes applications don’t use document level security, (Reader names fields), then when using 6.x GSA, select GSA Document ACLs for Document Authorization when registering the application. This will send Policy ACLs to the GSA which is faster than connector authorization.
  • With 7.0 GSA, the new ACLs work with reader fields. A database-level ACL is sent for all databases. A document-specific ACL is sent when there are Reader fields for a particular document.

Preparing the Lotus Notes client

The Notes Administrator must create a Lotus Notes User ID for the connector with:

  • READER ACCESS to all databases and documents that you wish to index. Specifically, where document-level access is restricted, ensure the Lotus Notes user for the connector is listed in the Reader names fields or is a member of a group or assigned a role that provides read access.
  • CREATE DATABASE permission on the Domino server that will host the connector database.
  • READER ACCESS to the Domino Directory for the domain.
  • MANAGER ACCESS with the Admin role to the GSA Connector Database
Tip: To avoid issues, we recommended that the Lotus Notes User ID be created with no password, and a certificate expiration at least 10 years in the future.

Before you install and configure the connector, apply the following change to the ID. In File -> Security -> User Security ensure that Automatically lock my Notes ID after x minutes is unchecked.


Installing the Lotus Notes Connector for the Google Search Appliance

The Lotus Notes Connector is installed with the Google Search Appliance Connectors installer.

  1. Launch the installer and accept the license agreement.
  2. Select a single connector type to install and select the GSA Connector for Lotus Notes.
  3. Select the Notes program directory and data directory for the connector.
  4. Enter the name and the port that you want to assign to the connector service, and the GSA IP address.
  5. Select the java.exe to be used by the connector. Remember that the connector requires a 32 bit Java 1.6 JDK.
  6. Select installation directory and program group name and the check the installation details before proceeding.
  7. You can register the connector manager with the GSA at this time or you can register this later manually using the GSA Admin Console. To register it now, provide the GSA administration username, password and port. Also provide a name for the connector manager.

Configuring the Google Search Appliance for Lotus Notes

To configure the connector for Lotus Notes you will need to perform the following:

  1. Create a connector database on the Domino Server
  2. Create the connector in the GSA Admin Console and point it to the connector database on the server.

Creating the Connector Database

You will require the assistance of the Lotus Notes Administrator to complete these steps.

Important: The Connector for Lotus Notes runs as a service on Windows. You must not have the Notes client started or start the Notes clients when the connector is running, otherwise resource sharing conflicts in the Notes API will cause the connector to crash. Before starting the Domino Administrator or client on the connector machine please stop the Notes connector service in Windows (default name LotusNotes1).
Step 1. Sign the GSA Connector template

The GSA connector template will be copied to the Notes client data directory during installation. The name of this template is gsa-connector_<version>.ntf. This template must be signed before it can be used.

The following instructions are for Domino Administrator 8.5. For other versions, consult the IBM documentation at Domino Administrator 8.5 - Signing a database or template.

To sign the database:

  1. Start Domino Administrator.
  2. Select File-Open Server-Local from the menu.
  3. Select the Files tab.
  4. Set Show me to Templates only.
  5. Find the template GSA Connector Template (version), right click it, and select Sign.
  6. Click OK.
Step 2. Create the GSA Connector database on the Domino server

To create the Connector Database:

  1. Start the Notes client.
  2. Select File->Application->New from the menu.
  3. For Server, select the Domino server that will host the connector database.
  4. Enter a Title and Filename for the connector database.
  5. In Template select the GSA Connector Template (Version). You may need to select Show Advanced templates.

    Creating the GSA Connector database
  6. Click OK.

    Note: Details about how to set up parameters in the GSA Connector database documents can be found in Configuring the GSA Connector Database.
Step 3. Configure the Lotus Connector for Google Search Appliance Database

You will now need to provide the initial configuration for the connector database.

Tip: If this is a migration from a previous release, copy information from the previous version of the configuration database into this one.
  1. Select File->Application->Open from the menu and select the connector database. The Setup document will open. Review the configuration and change the default values if necessary.
  2. Click on the Domino Servers tab.
  3. Click the Add Region button.

    Region
  4. In Region enter any name to uniquely identify this group of servers, for example MyRegion.
  5. In Domain specify the DNS subdomain for the Domino servers in this region starting with a period. For example .mydomain.com.
  6. Click on the button in Servers and select the Domino servers that are in this region. To select multiple servers, click in the left margin and a check mark will appear.
  7. Click Save & Exit to return to the Setup document.
  8. Click Save & Edit to save the connector setup and open the Connector database.
Important: Close the Connector database by clicking on the close button on the tab. This will force the Notes client to refresh the display.
Step 4. Register a Lotus Notes Application for Indexing

Every Notes database is based off a template or design. The Lotus Notes Connector is designed to use the template definition to make searches more intelligent and to avoid indexing unnecessary items.

Discovering the Template Name

Before you register a database or set of databases, you will want to know the template it is based on. To find this out:

  1. Right-click the database icon on the workspace and choose Application\Properties.

    Database Properties
  2. Click the design icon, which shows a straight-edge and triangle symbol.

    Database Design
  3. The template name is listed under inheritance. Generally template names reflect the type of template that has been used, for example, Help, Document, etc.
  4. Now, set up a sample database to crawl.
Specifying the Template
  1. Select File->Application->Open from the menu and select the connector database.
  2. Click New Database.
  3. Click on the button in Server and database to select a Domino application for indexing.
  4. Click OK.
  5. Click the button in Template to use and select the appropriate template. For DECS Installation and User Guide we recommend you use the Notes Help template. If you need a hint about the correct template, right click the database from the workspace and choose Properties.
  6. If this is the first time you are configuring the connector and you are indexing the DECS Installation and User Guide, Google recommends you set Document Authorization to No Security to make verifying the connector easier.
  7. Review the other defaults for the database and change them as preferred. For more information on database configuration see the Reference section.
  8. Click Save and then click Exit.
  9. Look in to one of the Domino servers you registered in a Region. To verify the connector is working, start with a generic database such as DECS Installation and User Guide in the help directory.
Note: For more details about setting up templates, see Creating Templates, Forms and Subform Definitions.
Step 5: Set up a Notes Connection

Before you install the Google Search Appliance Connector for Lotus Notes, you must make the following changes to the Google Search Appliance to accept and index feeds from the connector:

  1. Navigate to Connector Administration.

    Admin Console Connector Admin menu
  2. Edit the Connector Manager.

    Edit Connector Manager
  3. Enter the URL and port as shown. Use the Lotus Notes server’s IP address.
Step 6. Create the Connector in the Google Search Appliance

Follow the instructions to register the connector manager on the GSA if it has not already been registered.

To add a Lotus Notes connector:

  1. Ensure that Apache Tomcat is running.
  2. In the Google Search Appliance Admin Console, click Connector Administration -> Connectors.

    Admin Console Connector Admin menu
  3. Choose Connector Manager.
  4. Create a Connector Manager for Notes:

    Connector Configuration Lotus Notes
  5. In the Connector Name field, type the name of the connector instance and select Lotus_Notes as the Type.

    Each connector instance added to a particular connector manager or Google Search Appliance must have a unique name. The connector name must consist of no more than 64 alphanumeric characters. All alphabetical characters must be lower-case. Connector names may include underscores (_) and hyphens (-), but they cannot begin with a hyphen.
  6. On the Type drop-down list, select Lotus_Notes.
  7. Click Get Configuration Form.

    Connector Configuration Form Lotus Notes
  8. The connector manager name, connector name, and connector type are displayed.
  9. In the ID Password field, type the password for the Lotus Notes User ID for the connector. If the User ID has no password you can leave this field blank.
  10. In the Database field, type the path to the connector database on the server, for example gsa/gsaconnector.nsf.
  11. In the Server field, type the abbreviated hierarchical Domino Server name, for example, DominoServer1/Acme/Org
  12. If users will authenticate against something external, such as Active Directory, set the Notes Users Authenticate with Global Names property. Ensure that the formula used in the connector database to generate usernames will match whatever the user will type when logging into the GSA. The Notes connector has to map to the username sent from the GSA to a Notes user record using that value.

    Ensure that the GSA names created in the Notes username cache match the names the users will log in to the GSA with. Currently those names are generated using a Notes formula as the connector populates the username/group cache. Refer to a later section for details about setting this.
  13. Provide the GSA administration user and password. This is required only if you are using Policy ACLs to model database ACLs.
  14. To maximize crawl performance you can set the Traversal Rate to 2000 and Retry Delay to 1. This will allow the GSA to feed up to 2000 documents per minute to the GSA and scan for new updates every minute.
  15. When ready, click Save.
  16. Review and edit the connector manager settings.

    Edit Connector Manager
  17. Enter the URL and port as shown. Use the Lotus Notes server’s IP address.

Setting up Crawl Parameters

To configure crawl and feeds for the connector:

  1. In the Admin Console, navigate to the Crawl and Index > Crawl URLs page. In the Follow and Only Crawl URLs with the Following Patterns box, add the following statement:

    ^googleconnector://

  2. Save the configuration.
  3. Click Crawl and index > Feeds.

  4. In the List of Trusted IP Addresses section, select:
    • Trust feeds from all IP addresses or
    • Only trust feeds from these IP addresses.
    If you select Only trust feeds from these IP addresses, type in the IP addresses of the connector host.
  5. Click Save Settings.

Verifying and Troubleshooting the Connector

Once you create the connector in the GSA Admin Console, the connector will start to connect to the Domino server and validate the connector configuration in the GSA Connector Database.

To verify the connector is working correctly you should look in the logs. Look in the <INSTALLROOT>/<ConnectorInstance>/Tomcat/logs/google-connectors.0.log.

If you have configured the Notes client and the GSA Connector Database, and created the connector correctly in the GSA Admin console, then the connector should start building the group cache and people cache, and start traversing the DECS Installation and User Guide, which is the default database set up to help you validate all is working.

You should see messages similar to the following snippets:

Mar 15, 2012 12:42:22 PM [Config edt137_to_entyk02] com.example.enterprise.connector.instantiator.SpringInstantiator

setConnectorConfiguration

INFO: Start traversal

Mar 15, 2012 12:42:25 PM [] com.example.enterprise.connector.notes.NotesUserGroupManager updateGroups

INFO: Creating group BigGroup2

Mar 15, 2012 12:42:26 PM [] com.example.enterprise.connector.notes.NotesUserGroupManager updateGroups

If your application contains many documents, it can take several minutes or even hours to do the initial scan of the database. If you restart the connector, this scan will restart from the beginning. You can check on the connector status by clicking on the CrawlQ and SubmitQ buttons to see if documents are scheduled for crawling or sending to the GSA.

Enabling debug logs

To troubleshoot connector issues, you can enable debug logs in the GSA Admin Console or by editing the following file.

<INSTALLROOT>/<ConnectorInstance>/Tomcat//webapps/Connector-Manager/WEB-INF/classes/logging.properties

  1. Under Global properties change .level=INFO to .level=ALL or add the line:

    com.google.enterprise.connector.notes.level=ALL
  2. Restart the connector for the new log level to be applied.
Troubleshooting Scenarios and How to Solve Them
Documents do not appear as expected in the GSA

The two most likely causes of the document not being crawled are:

  • The Notes ID does not have access to the documents
  • The selection formula does not include the document.

To troubleshoot this:

  1. Check that the selection formula in the template includes the document(s) that is missing.
  2. Check that the Notes client has access to the documents that are missing by opening the document in the Notes client interface.
  3. Turn the logging level up to FINEST and set the maximum log size to a large size, for example 50MB.
  4. Get the document Universal ID of the document.
  5. Edit the document and wait for the connector to traverse the database again.
  6. Check the logs to verify that the document was included in the crawl.
The connector does not start

If the connector does not start, check for errors in the connector log. Common problems are:

  • The path to the configuration database is incorrect.
  • The name of the server is not fully qualified or incorrect, for example, the name should be similar to DominoServer/Berlin/Acme.
  • The system configuration document has not been created or contains incorrect values.
  • The password provided for the Notes User ID in the Administration console is not correct.
Fields do not appear as metadata in the documents indexed by the GSA

To map fields to be metadata in the GSA you must configure the template and configure the CustomMeta Fields. This is different from selecting the fields in a form to be included in the content.

The connector unexpectedly terminates or appears stopped in services

Be sure the Notes client is not running. The connector runs as a service using the system account. Since Notes uses shared resources, the Notes client and the connector cannot be running at the same time. Ensure that the Notes client is not running on the console or any Remote Desktop session.


Configuring the GSA Connector Database

Configuring the Setup Document

This section describes the configuration settings in the Setup document of the GSA Connector Database.

Domain Directory Filename

If the server that hosts the GSA Connector Database uses a different name for its Domino Directory specify the path here. Multiple Domino Directories are not supported.

Example: directory/directory.nsf

Using an external directory or Active Directory for authentication

If you use another directory system for authentication, then you will need to add users’ external userid to the Domino person document in the Domino directory if it does not match. Edit the system setup document and change the Formula to generate username so that the external userid is used. Then click Refresh Directory to update the people cache.

User Selection Formula

Normally all users listed in the Domino Directory are permitted to use the Google Search Appliance. If you would like to restrict this then you can add additional criteria to the formula.

If you change this setting, the directory cache will be rebuilt the next time traversal occurs.

Example: @Select Form="Person" & @Name( [OU1] ; FullName ) = "USA"

Formula to generate username

This configuration setting allows you to set the username (not to be confused with the Notes User ID) that users enter when they authenticate to the GSA. The default is the shortname in their person document.

When using another authentication system, such as LDAP, Active Directory or CA SiteMinder, you might want to map the username to be the same as their LDAP uid or email address.

Example: @LowerCase(shortname) + "@acme.org"

Domain Directory Cache Update Interval

The connector creates a cache of nested groups and people which is used to authenticate users, perform group resolution and authorize documents. This task runs in the background in the connector and by default runs once a day. Change this to run it more or less frequently.

Example: 360

(This will run 6 hours after the last update)

Prefix for GSA Policy ACL Groups

When indexing documents from multiple content systems or multiple Domino domains, there is the risk that group names will be the same between systems causing a collision and leading to unexpected authorization results. To avoid this, the Domino connector prefixes each group name in ACLs sent to the GSA and when resolving the group membership list for a user. Normally, there is no need to modify the default value unless you are running multiple Lotus Notes connectors and these connect to different domains in which case unique values must be used for each domain.

Example: AcmeDom

Database Categories

This entry provides allowed values for the dom_dbcategories meta tag. When a database is assigned to one of these categories, this value will appear in the dom_dbcategories meta tag for each of the documents in the database.

Example: HelpDesk; HR; Sales;

GSA record lock attribute

This determines if the “Lock” attribute is set for documents sent by the connector.

Example: Lock

Number of crawler threads

This connector can start multiple crawler threads to improve traversal performance. Typically, there is no need to change this value since overall throughput is limited by the GSAs indexing capability. In some cases where Lotus Notes applications are on remote Domino servers, additional threads can improve traversal performance. The maximum value for this is 5.

Example: 1

Maximum Crawl Depth Queue

To avoid creating a very long crawl queue and using a lot of temporary storage, the connector will stop polling for changes when this threshold is exceeded. The poller will include all new and changed documents that meet the selection criteria in the crawl queue when it polls a database, so when a new application is registered, the crawl queue can be much larger. As an example, if you register a new application with 30,000 documents with a selection formula of @All, when the poller processes the application 30,000 documents added to the crawl queue. After this, the poller will stop checking applications for changes until the crawl queue contains less documents than this setting (default is 5000).

Example: 5000

Deletion Check Batch Size

The connector checks the Indexed documents view for deletions each time traversal takes place. Once applications have been fed to the GSA, how often the deletion checking occurs is determined by the Retry Delay configured for the connector in the GSA Admin console. The deletion check process starts with the first document in the view and will check a number of documents up to this setting. When the next deletion check occurs, it will resume at the last document checked. To check for deletions more often, increase this number.

If you want to remove all documents from the GSA for a registered Domino application more quickly:

  1. Set this to a very high number.
  2. Open the database configuration document in the GSA Connector Database.
  3. Click Edit.
  4. Click Stop Processing and Purge.
  5. Click Save and then Exit.

Example: 300

Maximum attachment size

To reduce network utilization, you can omit attachments that are larger than this size from being sent to the GSA.

Example: 30

Attachment Spool Directory

The connector will extract attachments from documents as it traverses an application and save them temporarily to this location before they are sent to the GSA for indexing. The connector will stop processing if there is less than 300MB free in this directory. Enter a valid operating system path with read/write access for the connector or the following default will be used:

<Notes Client Data Directory>/gsaSpool

Windows Example: d:\gsaSpool

Linux Example: /var/gsaSpool

File type exclusion

The connector will skip processing files with extensions in this list. If you want to exclude attachments with no extension, enter the period character.

Example: a; afm; ai; arj; avi; bin; bmp


Configuring the Database Registration Document

This section describes the configuration settings in the Database registration document of the GSA Connector Database.

Example: No Security

(Documents will be sent to the GSA will be marked public and visible to all users).

Categories

When a database is assigned to a category, this value will appear in the dom_dbcategories meta tag for each of the documents in the database sent to the GSA.

Example: Help; HR

Update Frequency

To control how often a database is checked for new and changed documents you can change this setting. The default is 1 which means typically every time a traversal occurs, the database will be checked for changes.

Example: 1

Lock records into index

This will determine whether documents from this database will have the lock attribute set.

Example: True

Check for deletions

The deletion check task will not check documents from this database if this setting is set to false.

Example: True

Replica servers

The values in this setting will appear in the dom_servers meta tag for each of the documents in the database sent to the GSA.

Example: Domino2; HubServer/Acme/Org


Creating Templates, Forms and Subform Definitions

The Google Search Appliance for Lotus Notes includes a small set of templates for common Lotus Notes applications to help you get started quickly with indexing Domino applications.

For custom Domino applications, you may want to change which documents in an application are sent to the GSA or set the document titles to improve the quality of search results. To do this, the connector provides templates. These allow you to:

  • Filter which documents in an application are sent to the GSA for indexing.
  • Determine which fields in a document are sent to the GSA for indexing.
  • Set the Title and Description meta tags for a document. Send additional Domino document fields as meta tags to the GSA.

Before you start

Before creating or modifying a template, you will need the following skills and hence likely the assistance of the Notes Administrator or application developer in your organization:

  • Domino application development and design elements
  • Knowledge of Domino @Formulas
  • Familiarity with the Domino application to be indexed

Templates

Adding a template

To add a template to the Connector database, complete the following detailed steps:

  1. Click Templates.
  2. Click New Template. You will be prompted to enter the template name.
  3. Enter the name of the template and click OK. The system checks to make sure that the name has not been used before. If the name is not unique you will not be able to create the new template.
Reassigning a template

The Connector template definition documents read the design of the database originally chosen when the template document was first created to get form, sub-form and field details to assist in the completion of template information.

Occasionally you may need to modify a set of template parameters but the original database is no longer available for reference. In such instances, you can re-assign the template profile to a new database.

Should you ever need to do this however, you should be sure that both the old and new databases share a common design.

To reassign a template to a new database, complete the following steps:

  1. Select the Templates option from the menu.
  2. Locate the desired template and double click to open the document.
  3. Click Edit.
  4. Click Change Source Database and click Yes to the warning prompt.
  5. In the Open-Database dialogue, locate and select the new database on one of your servers and click Open.
  6. Click Save in the action bar button.

You can now make changes to the template parameters as required.


Custom Meta fields

This configuration variable allows the contents of a document field to sent as metadata to the GSA. This accepts three formats.

fieldname If the connector encounters a field called fieldname in any document, the contents of the field will be sent to the GSA in a metadata element with name=fieldname.
fieldname=custommetaname If the connector encounters a field called fieldname in any document, the contents of the field will be sent to the GSA in a metadata element with name=custommetaname.
formfilter===fieldname=custommetaname If the connector encounters the field called fieldname in a document where the field Form has the value formfilter, the contents of the field will be sent to the GSA in a metadata element with name=custommetaname.

Example: Person===shortname=uid; Group===listname=uid

In this example, for documents with Form="Person", the contents of shortname will be placed in a meta tag uid and for documents with Form="Group", the contents of listname will be placed in the meta tag uid.

The fieldname or custommetaname must be less than 30 characters in order for the metadata to be added to the document.

Form Definitions

Form definitions enable you to modify which fields in a document are sent to the GSA for indexing and to modify the Title and Description formulas if desired.

In version 2.8 and higher, form definitions are not required. If the connector encounters a document with no form defined, it will send the contents of all fields to the GSA and use the Title and Description formulas from the Template definition.

To add a form definition for a template:

  1. Open the Template.
  2. Switch to the Form Details tab.
  3. Click Add Form Definition.
  4. Select the form from the list and click OK.

More about Fine-tuning the Search with Notes Templates

Note: You will need to have Domino Administrator access to carry out these tasks.

Most crawlers for Notes only support full content crawling of a Notes database. Invariably however, a Notes database will contain documents and fields that are used only to support the Notes application itself. This information adds no value to the search experience and as such should be excluded from the search index. The Google Search Appliance Connector for Lotus Notes excludes this meaningless information through the use of its template profiles.

Without an understanding of the design of the Notes database it is difficult to know what information is of value and what isn’t and therefore the normal approach to building a searchable index is to crawl and extract every item on each Notes document.

Every Notes database that is registered with the Connector must be assigned to a template profile. Templates tell the system which documents and fields to select as part of the content extraction process for the GSA. They also tell the system how to build key metadata values such as document title and description.

In this way, you can control exactly which information is extracted form your Notes databases for inclusion at the GSA. The diagram below shows how template information is used to determine what information is extracted from each database that has been registered with the Connector.

This approach has three main benefits. These are:

  • Templates allow you to control exactly what Notes information is sent to the GSA.
  • Form profiles allow common metadata values prior to submission to the GSA.
  • Document selection ensures that only documents of value are sent to the GSA. Any document that falls outside of the criteria specified after inclusion in the index will be removed from the GSA just as if it had been deleted from the source database.

Sample templates are provided with the connector for the more common types of Notes database such as Discussion, Team room, Document Library, Office Library, Mail files, Document Manager etc. For the lazy, there is even an “All documents and fields” template. These can be used as is or modified as required. You can also define your own profiles for any custom applications as required.

Although the Google Search Appliance Connector For Lotus Notes does support full content crawling through it’s All documents and fields template, the Connector also offers the far more effective selective content crawling by using template profiles to describe each type of database that will be crawled. These profiles tell the crawler which documents and fields to select as part of the crawl process for each assigned database.

In addition to defining the documents and fields to include in the index, templates in the Connector database are used to define the key meta data values that are passed to the GSA for each Notes document. These are the document title and document description values.

Using Templates to Leverage Metadata

The Connector provides a facility to make any Notes field a metadata item in addition to the default values shown above. You can then use this information in search results, search forms, and to support field level search of your Notes content from the GSA using the advanced Google search operators.

By carrying the common metadata from the host Notes document with each attachment record users can discover attachments from the parent document’s metadata as well as through the attachment content itself.

By including link information in the metadata it is possible to offer the following sort of functionality from the search results:

  • Open any attachment from any Notes document found in the result set
  • Open parent document from any attachment
  • Open any other parent document attachment from an attachment in the search results
Adding a Template to the Database

If you wish to extend the provided templates with your own, you can do so by adding your own templates. To add a template to the Connector database, complete the following steps:

  1. Add a template document.
  2. Add form definition documents.
  3. Add-sub form definition documents (optional).
Adding a Template Document

To add a template to the Connector database, complete the following detailed steps:

  1. Select Templates from the menu.
  2. Click the New Template button. You will be prompted to enter the template name.
  3. Enter the name of the template and click OK. The system will check to make sure that the name has not been used before. If the name is unique, the new template form will be displayed, otherwise you will be asked to try again.

Complete the Template Details tab as follows:

  1. Enter a brief description of the template.
  2. Using the button provided choose a database that will be used as the source of design information for this template. Please note the following points:
    • The system uses the chosen database as the source of its form and sub-form lists and field information.
    • The databases that you assign to this template later should share a common design with the database you choose here for the template information.
    • The database you choose here provides design details only; it is not by default indexed by the system.
    • You can choose to index the chosen database if you wish by adding a database parameters document later.
  3. Enter the selection formula to define the documents that should be selected for inclusion in the index. The formula entered here should be written in the same way as a Notes view selection formula; for example, if you wanted to index all Person documents in the Domino Directory (and only Person documents) you would enter:

    @Select Form = "Person"

The system can optionally automatically generate summary documents. Summary documents will be created automatically for any document that is covered by the selection formula entered above which do not have a form definition document defined. The default setting is “No” and to ensure best performance and meaningful results it is recommended that this setting remains unaltered.

Auto-generate rules

If the template being defined represents databases that meet both of the following rules, you can use auto-generation, and there is no requirement to add any form definition documents to the template.

  • You require ALL fields to be indexed for all documents selected by the template selection formula.
  • You will use field names for the generation of document titles and descriptions (rather than formulas).

To provide greater control over index contents, Google recommends that the auto-generate feature be disabled in favor of defining a definition document for each form that is covered by the document selection formula.

Reassigning a template

The Connector template definition documents read the design of the database originally chosen when the template document was first created to get form, sub-form and field details to assist in the completion of template information.

Occasionally you may need to modify a set of template parameters but the original database is no longer available for reference. In such instances, you can re-assign the template profile to a new database. Should you ever need to do this however, you should be sure that both the old and new databases share a common design.

To reassign a template to a new database, complete the following steps:

  1. Select the Templates option from the menu.
  2. Locate the desired template and double click to open the document.
  3. Press the Edit button to take the document into edit mode.
  4. Press the Change Source Database button, and answer, Yes to the warning prompt.
  5. From the Open-Database dialogue, locate and select the new database on one of your servers and press the Open button.
  6. Press the Save action bar button.

Now you can make changes to the template parameters as required.

Note: To ensure that system integrity is maintained at all times, it is only possible to create and edit template documents and their associated form and sub-form definition documents from the administration server.

Frequently Asked Questions

How are documents sent to the GSA?

Documents are initially scanned from the source database using the selection formula and queued for crawling. You can check which documents are being queued by looking in the CrawlQ view. Once all the documents are queued for crawl, another thread will start retrieving the documents and processing attachments and queue these so they can be fed to the GSA. You can check the status of this by looking in the SubmitQ view. Attachments are temporarily saved on the connector machine in the local filesystem. The connector will then feed these to the GSA.

Why does the document count not match the number of documents in the Notes database properties?

The connector will send one document for each Lotus Notes document in the database that matches the selection formula. It will also send one document for each attachment that is not in the exclusion list or is not greater than the maximum attachment size. Hence, the numbers of documents in the database may not match the number of documents sent to the GSA. This is particularly important when indexing a database with a lot of attachments, as the document count may be much higher than expected.


Known Issues and Limitations

The following sections describe known issues and limitations, as well as workarounds by component:

  • Domino Administrator is not available on Linux so you cannot sign the GSA Connector Template.
  • You must transfer the GSA Connector template to a Windows machine, sign the template using the Domino Administrator and then transfer the signed template back to the Linux connector host. Domino server names must be valid DNS names.
  • A-Z 0-9 and - and _ are the only supported characters.
  • ACLs only support Databases
  • Reader fields are not supported.
  • Use connector security if document level security is required.
  • 64K limit to nested group sizes. Nested group membership levels are restricted to 64K of data.
  • Only a single Domino Directory is supported regardless of the number of connectors used.
  • Directory Catalogs are not supported.
  • Linux is not a supported platform.
  • Use a supported Windows client for the connector.
  • Server Access is not supported.
  • The server access controls in the Domino server document are not enforced by the connector.
  • Group names with the forward slash character (/) are not supported.
Was this article helpful?
How can we improve it?