This article describes data processing filters, which are applied before reports are generated. In addition to data processing filters, Urchin provides report filtering on the reporting interface. Read Reporting Interface --> Report Side Filtering for information.
To create a filter, click the Add button in the Filter Manager screen.
Filtering Sequence
Each time the scheduler runs a profile, each entry in the log files passes through the steps shown in the figure below. Before any of the report tables are updated, the 'raw' fields in the log file entry are parsed, which creates a number of 'auto' calculated fields. For example, the browser and platform fields are calculated from the raw cs_useragent field.
Filtering is applied once all of the fields have been populated, and before any entries are made in the report tables. Filters can be applied to any type of field, including calculated fields. No additional parsing occurs after filters are applied. Thus, it is important to apply the Filter to the correct field. A list of the purpose of each available field is provided in the next section.
Filters are applied in the following order:
- Advanced Filters, Search & Replace Filters, and DynamicURL Filters
- Decode URL and Japanese Encoding Filters
- Lookup Tables
- Include and Exclude Filters
Filter Types
- Exclude Pattern: This type of filter excludes log file lines (hits)
that match the Filter Pattern. Matching lines are ignored in their entirety; for
example, a filter that excludes Netscape will also
exclude all other information in that log line, such as visitor, path, referral, and
domain information.
- Include Pattern: This type of filter includes log file lines (hits)
that match the Filter Pattern. All non-matching hits will be ignored and any data
in non-matching hits is unavailable to the Urchin reports.
- Decode URL: This is a predefined filter that decodes URL-encoded
characters back to their original form. For example, '%20' in a URL is replaced
with a space. Apply this filter to URI-stems and queries to see the original text.
- Japanese Encode (UTF-8): This is a predefined filter, generally applied to
the keywords field or other potentially multi-encoded field, that looks for
Japanese
encoded words and converts the encoding to UTF-8 format for a consistent storage and
display.
- Search & Replace: This is a simple filter that can be used to search for a
pattern within a field and replace the found pattern with an alternate form. See the
section on Search & Replace Filters for more information.
- Dynamic URL (deprecated): This type of filter is used to translate arcane
dynamically generated URLs into more human-readable page names. Note: the new Page
Query
Terms Report duplicates the bulk of this function, and the Advanced Filter
encompasses all DynamicURL's features and more. It is strongly recommended that you
either eliminate old Dynamic URL filters if possible or else convert them to one of
the newer forms of filter.
- Advanced: This type of filter allows you to build a field from one or two other fields. The filtering engine will apply the expressions in the two Extract fields to the specified fields and then construct a field using the Constructor expression. Read the Advanced Filters article for more information.
Choosing Where To Apply a Filter
Filters can be applied either to profiles or to individual log sources. The scope of the filter can be different for each of these cases. A filter applied to a profile will affect all log sources processed for that profile. A filter applied to a log source will always affect that specific log source, even if multiple profiles are using the same log source. In general, you should apply filters to the profile unless one of the following cases occurs:
- You have multiple log sources for a profile and you do not want the filter to apply to all of the log sources.
- You have multiple profiles using the same log source, and you want all of the profiles to use the same filter.
Creating and Managing Filters
In the Urchin administration interface, click Configuration, then Urchin Profile-->Filter Manager. Click the Add button to launch the Filter Wizard.
Important Note: Filter patterns must not be longer than 255 characters!
Once you have created a filter, edit the profiles or log sources to which you wish to apply the filter, and add the filter.
To create a filter while editing a profile or log source, click the Profile Filters tab or Log Filters tab. A window appears showing the currently active filters. Click the Add button on this window to launch the Filter Wizard.
The filter creation screen has a dropdown menu at the top with selectable built-in filters for common filtering tasks such as filtering out robot traffic to your site. These built-in filters also serve as examples of how to set up various kinds of filters.