Importing, Exporting, and Scripting the Configuration


Urchin 4 provides utilities and functionality to allow the administrator to script all operations. Only the report viewing requires the web-based interface to be operational. All configuration and log processing can be scripted using the following utilities and techniques. For first time users, it is helpful to run the web-based administrative interface first, in order to get familiar with the terminology and capabilities of the system.

Urchin 4 includes four utilities for modifying the configuration database without using the web-based administrative interface. Located in the util directory of the distribution, the four utilities are:


	uconf-export [-f file]
	uconf-import [-r|-o] [-f file]
	uconf-driver [-f file] [-d directory]
	uconf-schedule [-r]

Each of these utilities is called from a command prompt. A summary of usage:


  1. The uconf-export utility will export the entire configuration into a file or to stdout, if no file is specified. The format of the exported data is an XML type format defined later in this document. Each record in the exported data corresponds to a configuration record in the configuration database.
  2. The uconf-import utility can import the same XML type formatted data into the configuration database. This can be used to import or edit single records or to replace the entire configuration with the contents of a file similar to that found in Urchin 3. The "-r" option instructs the uconf-import utility to remove the entire configuration before importing. The "-o" option instructs the uconf-import utility to overwrite existing duplicate records with the information provided.
  3. The uconf-driver utility performs specific actions to individual records. All parameters can be passed on one line as arguments to the script, or a file with multiple commands (one per line) can be used. The "-d" option can specify a different location of the configuration database than the default relative path.
  4. The uconf-schedule utility invokes global scheduling changes for all configured profiles, including the ability to run all profiles immediately. When invoked with the "-r" option, uconf-schedule will schedule all profiles to run immediately via the Urchin Scheduler without any further prompting. The Urchin Scheduler urchind must be running for this utility to either execute jobs immediately or schedule them for future execution.

In addition to using the global scheduling utility, the operation of Urchin log processing can be triggered by calling the Urchin engine directly, specifying the name of the profile on the command line. Located in the "bin" directory of the distribution, the Urchin engine can be run as follows:


   urchin [-H] -p profilename

The name of the profile should be called within quotes if it contains any white space. The "-H" option tells the Urchin engine to log all output into the history record which is accessible through the Urchin admin GUI interface. Otherwise all output is written to stdout.

Note that you can use the uconf-export and uconf-import utilities to easily make a backup of or restore your Urchin configuration. This provides a very quick method of recovering an Urchin installation after a disk failure or other system problem. Examples of this functionality:


  • Save configuration for safekeeping:
    uconf-export > /path/to/saved-configurations/urchin-
  • Restore Urchin configuration from a known good backup:
    uconf-import -r -f /path/to/saved-configurations/urchin-

Intended Usage

The uconf-export and uconf-import utilities are intended to replicate the functionality of the Urchin 3 text-based configuration and to provide a simple method for importing and exporting data from the Urchin configuration database to and from static configuration files. These utilities also allow you to specify the names of profiles, log sources, filters, users, etc. in access lists rather than the more cryptic record number lists that are used by the uconf-driver utility. The uconf-import utility can be used to add new records or modify existing records, but it cannot remove old records (unless the database is completely reset with the "-r" option).

The uconf-driver utility is very powerful and can be used for very specific scripting operations that may change only a few parameters in a database record, as well as performing complete record additions, modifications and deletions from the database. It can also be used for querying the configuration database for several parameters. This utility is more ideally suited for use in an environment where scripting all administrative functions of Urchin is desired, such as in automated provisioning systems or very large hosting environments where use of the Urchin administrative GUI is impractical.

It should be noted that both the uconf-import and uconf-driver utilities allow arbitrary name/value pairs to be entered into the database. This allows for both forward and backward compatibility within Urchin, and may also be useful for storing site-specific data such as a user id or other identifier. However, supporting this functionality also has the side effect that neither utility does any checking to ensure that only valid Urchin name/value pairs are entered into the database. It is the responsibility of the administrator to ensure that data is entered properly.


When called from the command line, the uconf-export utility will export the entire configuration into a human-readable, text format. The output is written to stdout unless a file is specified using the "-f" option. The format of the exported data is a multi-line format. Each group of lines corresponds to one record in the configuration database. The group of lines is indicated by begin and end tags in the form:


  <table name="record-name">

where "table" is one of the possible table names within the configuration database (i.e. Profile, Logfile, Filter, etc.) and "record-name" is the name of the record. Between the begin and end tags, a list of configuration parameters that define the specifics about that record are listed as name/value pairs, one per line. Any line that begins with a "#" should be treated as a comment and not part of the configuration data.

The uconf-export utility can be used as a snapshot and backup of the configuration data, or it can be used in conjunction with the uconf-import utility to modify the existing configuration. For a complete list of the different tables and configuration directives for each, please see the reference tables at the end of this document.


When called from the command line, the uconf-import utility will read the contents of the file specified by the command line "-f" option or stdin, if no file is specified. The contents of the file should contain one or more configuration records in the same format that is created with the uconf-export utility. Each record should begin with the appropriate begin tag and end with the appropriate end tag. Each record should contain all of the desired configuration directives as name/value pairs on separate lines between the begin and end tags. Leading and trailing white spaces are ignored. The following example illustrates a very basic entry for a user:


<User Name="bob">   
   ct_fullname=Bob Johnson

In this example, the record is for the User table and the name of the record, which is the username, is bob. Additional parameters are provided on the next three lines each beginning with a configuration directive, an equals sign and then the value of the directive. Note: plaintext passwords will be automatically encrypted on insertion unless they are already encrypted as indicated by the "USCC" designation. This allows you to add new users or edit existing users without being able to export the original passwords.

By default, the uconf-import utility will not overwrite or delete any existing records. To simply add new records, create them in a file and use the utility with only the "-f" option. The "-r" option instructs the utility to remove all existing records from the configuration database before importing, including the global configuration that contains licensing information, machine settings and the admin password. This option should be used with caution! You can use the export utility and then import the entire configuration with the -r option to replace all records. Important! When using the -r option with uconf-import, it is required that the first three records in the imported configuration be as follows. It is also highly recommended that you include the (admin) User record to ensure that you retain the correct password and preferences for the main administrator account:


<Global Name="Access Settings">
<Machine Name="Process Settings">
<Affiliation Name="(NONE)">
<User Name="(admin)">

Using uconf-import with the -o option allows for the addition of new records and editing of existing records. This option will not delete any previously existing records, but it will overwrite any records with the data found in the specified file. When editing a record, you must specify all of the desired configuration directives for that record, not just the ones you are changing. The uconf-import utility completely replaces entire records and will not incrementally change parameters within a record. For a complete list of the different tables and configuration directives for each, please see the reference tables at the end of this document. An easy way to get familiar with using the import functionality is to first use the admin interface to create some records for Profiles, Log Sources, Users, etc., and then use the uconf-export utility to create examples of what the configuration should look like.


When called from the command line, the uconf-driver will read the contents of the file specified with the -f option or stdin, if no file is specified. The uconf-driver will also read the argument list allowing a single command to be passed via the argument list. Each line of the file specified or stdin should represent one command. The format of the command is a series of name/value pairs that define the command to be taken. In particular, the ?action? variable specifies the specific command to take. The nine possible actions are:

action=nrecords table=table
action=list table=table [start=i] [n=j]
action=get <id>
action=add table=table name=name [cx_xxx=value cx_xxx=value ?]
action=edit [cx_xxx=value cx_xxx=value ?]
action=delete <id>
action=get_parameter <id> parameter=cx_directive
action=set_parameter cx_directive=value

where <id> is one of:

table=table entry=i
table=table name=name

A single action can be accomplished via the argument list or a series of actions can be specified in a file. The table variable specifies one of the Table Names listed in the Table Name Definitions in the next section. The name variable specifies the exact name of the record. Specific configuration directives begin with "ct_", "cr_", or "cs_". A complete list of directives for each table is specified in the next section.

Upon completion of a command, the uconf-driver will print out one of the following based on the action parameter that was specified:


Command Output Description
(all) [usage msg] command line not in recognized format
(all) [no msg] command didn?t perform any action, perhaps due to an out of range entry, incorrect table name, etc.
(all) -l command line parameters invalid for request type
add [recnum] record number of record that was created
delete [recnum] record number of record that was deleted
edit [recnum] record number of record that was edited
get [record] complete set of name/value directives for specified record
get_paramenter [param] value of requested directive
list [records] complete list of all name/value directives for all records in the specified table
nrecords [count] count of records in the specified table
ntotalsrecords [count] count of all records in all tables
set_parameter 0 always prints 0 on success

Several of the commands require you to identify (<id>) the record that is to be retrieved or modified. This identification can be accomplished in three ways. If the internal record number is known, then the ?recnum? variable can be used. This number is not necessarily sequential within a table. If it desired to loop through all of the records in a particular table, then the ?entry? variable can be used in conjunction with the table name. The first entry is 1. And finally, the exact name of the record can be specified using the ?name? variable. Be sure to use quotes if the name may contain white space.

The "ntotalrecords" command outputs the total number of records in all tables in the configuration database. The "nrecords" command outputs the number of records in a single table, specified using the table variable.

The "list" command will retrieve multiple records from a particular table. Each record is printed on a separate line. Optionally, the ?start? variable specifies which entry to begin grabbing, and the ?n? variable specifies how many. For example, if a particular table has 44 records and you want to grab records 20-29, set start to 20 and n to 10.

To grab a single record, the "get" command takes the provided identifier, and prints all information associated with the particular record on a single line. As mentioned above, the identifier can be provided in three ways.

The ?add?, ?edit? and ?delete? functions provide comprehensive editing ability of records. The "add" command requires both the table and name of the record to add. A directive list should also be passed along with these variables. The "edit" command is similar to the add command, except the record directives are replaced with the new list of directives. This function clears all previous directives and adds all new directives specified in the command. The delete function deletes the record from the table. These functions will print the ?recnum? value on success and a ?1 on failure.

The "get_parameter" and "set_parameter" commands allow you to set and retrieve an individual directive from a record. In addition to the identifier, the get_parameter command requires the use of the parameter variable, which specifies the specific directive to look for. The set_parameter command can be used by passing the directive and its value as a name=value pair. Here is an example with two commands that add a user and then sets the default language for the Urchin reports to English:


	action=add table=user name=?bob? ct_fullname=?bob J? 
	action=set_parameter table=user name=?bob? cs_language=en 

These two commands could be passed one at a time as arguments on the command line or in a file. The password will automatically be encrypted. Be sure to use quotes around strings that may contain white space.

Note that the uconf-driver is a lower-level utility that does not map any of the directives for you. In using the driver to script some operations, please be aware that many of the tables contain directives that refer to other records, or lists of records. These directives are: ct_ulist, ct_glist, ct_llist, ct_flist, and ct_rlist, which refer to the user, group, logfile, filter, and profile tables respectively. These lists are represented as exclamation point delimited list of recnums:



where each entry represents the recnum value of a record and is surrounded with exclamation points. Important! Be sure to keep the cross-references intact. For example, a Filter record has a ct_rlist which details all of the profiles that the filter applies to; and a Profile record has a ct_flist which details all of the filters that apply to this profile. Important! The uconf-driver uses this exclamation point delimited lists of record numbers, whereas the uconf-export and uconf-import utilities use comma-delimited lists of names. Be sure to use the appropriate list specification depending on which utility you are using.

The uconf-import and uconf-export utilities translate and verify these lists for you, making their use a little easier than the uconf-driver.

Urchin Database Table Reference

The following tables provide exact details on the table names, directives, and meanings for each table. The first table outlines each of the table names and what it is used for. Then, for each table, a comprehensive list of directives is provided. Please note that each record does not need to contain every directive applicable to that record type, as some directives may not be applicable to that record. Urchin will use the default values if there is no explicit definition for a directive. Where applicable, default values are printed in bold typeface.

Table Name Definitions


Table Name Meaning/Purpose
Global General settings including licensing and remote access
Machine Process settings including DNS and memory usage
FIlter Specifies log and profile runtime filters
Logfile Specifies the location and format of a log source
Profile Meaning/Purpose
Table Name Report settings for a particular website
Task Runtime schedule for a particular profile
Affiliation Definition for enterprise-level control of groups and users
Group Definition for controlling a group of users access
User An individual user with login and password

Global Directives


Directive Meaning/Purpose
cr_dcmode datacenter mode (on|off)
cr_remoteaccess allow remote access (on|off)
cr_remoteadmin remote administration available (on|off)
ct_license license code used by Urchin Licensing
ct_name [internal use only]
ct_port port that apache runs on (default is 9999)
ct_serial serial code used by Urchin Licensing
ct_schedulers [internal use only]
cr_setupwizard run setup wizard first time (on|off)
ct_var VAR code used by Urchin licensing

Machine Directives


Directive Meaning/Purpose
cr_processdns turn dns processing on and off (on|off)
cr_priority run priority of urchin (low|normal|high)
cs_preset [internal use only]
cs_dnslops number of dns loops (default is 5)
cs_dnstarget percent target for dns (default is 80)
cs_limitdbtable # of records to limit per table storage (default is 10000)
ct_dbuffsize data buffer size in MB (default is 13)
ct_pbuffsize path buffer size in MB (default is 1)
ct_name [internal use only}
ct_resolverip resolver ip (default is
ct_sbuffsize session buffer size in MB (default is 3)
ct_tbuffsize text buffer size in MB (default is 1)
ct_vbuffsize visitor buffer size in MB (default is 2)

Filter Directives


cr_action [internal use only]
cr_filtertype type of filter (include|exclude|dynamicurl)
cs_filterfield field in log file to apply filter to (1-18) *
  1 = Date (IIS)
  2 = Time (IIS)
  3 = Date and Time (Apache)
  4 = IP Address
  5 = Username
  6 = Request (Apache)
  7 = Method (IIS)
  8 = URI Stem (IIS)
  9 = URI Query (IIS)
  10 = Status
  11 = Bytes
  12 = Hosts
  13 = User-Agent
  14 = Cookie
  15 = Referrer
  16 = Custom
  17 = Custom
  18 = Custom
cs_llist exclamation-point delimited list of log source recnums to which this filter is applied (uconf-driver)
  comma delimited list of log source names to which this filter is applied (uconf- import)
cs_rlist exclamation-point delimited list of profile recnums to which this filter is applied (uconf-driver)
  comma delimited list of profile names to which this filter is applied (uconf-import)
ct_affiliation optional affiliation
ct_filter filter expression (supports regular expressions)
ct_name descriptive name for filter

Logfile Directives


Directive Meaning/Purpose
cr_action [internal use only]
cr_decodeurl decode log url (on|off)
cr_logdestiny disposition of log after processing (1=don?t touch, 2=archive/compress, 3=delete)
cr_protocol remote log transfer protocol (ftp|http)
cr_type location of log (local|remote)
cs_flist exclamation-point delimited list of filter recnums which are applied to this log source (uconf-driver)
  comma delimited list of filter names which are applied to this log source (uconf- import)
cs_lf[1-20] [internal use only]
cs_logencoding logfile encoding type (1=English, 2=Japanese)
cs_logformat logging format for logfile (custom|ncsa|w3c) (default is w3c for Windows, ncsa for all others)
cs_rlist exclamation-point delimited list of profile recnums using this log source (uconf- driver)
  comma delimited list of profile names using this log source (uconf- import)
ct_affiliation optional affiliation
ct_loglocation local log pathname/location (e.g. /logs/access.log)
ct_logmap pipe delimited list of custom log fields mapped to Urchin?s internal notion of fields
ct_name identifier for log source
ct_password password for remote log access
ct_port port number (e.g. 21 for ftp, 80 for http)
ct_querytoken change the query token (? by default)
ct_remotelocation remote log pathname/location
ct_separator single character field separator character (\s, \t are escaped characters for space and tab)
ct_server server name or IP address
ct_username username (default is "anonymous")

Profile Directives


Directive Meaning/Purpose
cr_logtracking turn log tracking (on|off)
cr_processdns Process DNS (on|off) (requires that dns be enabled in the machine table)
cr_processpath turn visitor tracking (on|off)
cr_sessionpageview session requires a pageview (on|off)
cs_flist exclamation-point delimited list of filter recnums applied to this profile (uconf-driver)
  comma delimited list of filter names applied to this profile (uconf-import)
cs_glist exclamation-point delimited. list of group recnums granted access to this profile (uconf-driver)
  comma delimited. list of group names granted access to this profile (uconf-import)
cs_llist exclamation-point delimited. list of log source recnums associated with this profile (uconf-driver)
  comma delimited. list of log source names associated with this profile (uconf-import)
cs_pathlevel depth of path reporting (default is 3)
cs_processlevel number of level of path to report (default is 4)
cs_referrallevel referral level to report (default is 3)
cs_sidfield session id field (1-18) same list as filter.
cs_taskid recnum for associated task in Task table (uconf-driver only)
cs_timeoffset Time offset (in seconds) for data in log (default is 0=GMT)
cs_ulist exclamation-point delimited list of user recnums granted access to this profile (uconf-driver)
  comma delimited list of user names granted access to this profile (uconf-import)
cs_vmethod visitor tracking (0=IP-UserAgent, 1=Session ID, 2=UTM, 3=IP- Only)
cs_visitortimeout session timeout in seconds (default is 3600)
ct_affiliation optional affiliation
ct_defaultpage default page for site (e.g. index.html)
ct_lasthit [internal use only]
ct_name identifier for profile
ct_reportdomains comma delimited list of site domains
ct_sidpre precursor to session id pattern being matched
ct_sidpost terminator to session id pattern being matched
ct_website website URL

Task Directives


Directive Meaning/Purpose
cd_btime [internal use only]
cd_etime [internal use only]
cd_lastrun [internal use only]
cr_dow day of week to run task [0-6]
cr_enabled [internal use only]
cr_frequency task frequency
cr_runnow [internal use only]
cs_dom day of month to run task [1-31]
cs_hour hour of day to run task [0-23]
cs_minute minute of hour to run task [0-59]
cs_rid recnum for associated profile in Profile table (uconf-driver only)
ct_affiliation optional affiliation
ct_application [internal use only]
ct_completed [internal use only]
ct_day day of month to run task [1-31]
ct_lockid [internal use only]
ct_pid [internal use only]
ct_name task identifier name
ct_runstatus [internal use only]
ct_status [internal use only]
ct_year year for task to run [CCYY]

User Directives


ct_affiliation affiliation for user
cs_adminlevel admin level (1=admin, 2=affiliate admin, 3=user)
ct_fullname full name of user
cs_glist exclamation-delimited list of group recnums the user belongs to (uconf-driver)
  comma delimited list of group names the user belongs to (uconf-import
cs_language two-letter report language code for user
cs_rlist exclamation-point delimited list of profile recnums the user has access to (uconf-import)
  comma-delimited list of profile name the user has access to (uconf-import)
cr_leveltype affiliation admin privilege level
  0=manage users/groups/tasks
  1=manage users/groups/tasks/filters
  2=manage users/groups/tasks/filters/log sources/profiles
ct_name identifier for user
ct_password user password

Group Directives


cs_rlist exlamation-point delimited list of profile recnums the group has access to (uconf-driver)
  comma-delimited list of profile names the group has access to (uconf-import)
cs_ulist exlamation-point delimited list of user recnums assigned to the group (uconf-driver)
  comma-delimited list of user names assigned to the group (uconf-import)
ct_affiliation optional affiliation
ct_groupdesc description of the group
ct_name identifier for the group

Special Usage Notes

  • If a Profile is added but no corresponding Task is added, scheduling of the Profile cannot be managed within the Urchin admin GUI interface.
  • When defining a Custom Log format, the ct_logmap directive must be used to specify the mapping of the logfile fields in the custom log to Urchin's own internal representation of these fields. Please use the following table for constructing a logmap entry (cs_logmap directive):

Urchin Internal Log Format Representation


1 W3C date format : CCYY-MM-DD
2 W3C time format HH:MM:SS
3 NCSA time/date format [DD/Mon/CCYY:HH:MM-HH:MM]
4 Visitor IP address (IP only - no hostnames allowed)
5 Username
6 NCSA-style request (method<space>requested-page<space>
7 W3C method (e.g. GET or POST)
8 W3C URI stem
9 W3C URI query
10 status code
11 bytes transferred
12 visitor hostname (may be an IP address or a hostname)
13 User Agent string (browser and platform)
14 cookie list
15 referral
16 [custom field #1]
17 [custom field #2]
18 [custom field #3]
19 virtual hostname of server

For instance if your webserver produces logfiles with the following fields separated by spaces:

username IP NCSA_date NCSA_request referral status bytes

the corresponding cs_logmap directive would look like:


Examples Using uconf-driver and uconf-import Utilities

This section illustrates a number of ways that the uconf-driver utility can be used to perform different administrative tasks within Urchin. Note that Urchin defaults will be used for parameters that aren?t explicitly specified. In the examples below, line breaks are added for readability.

Add a user (note that the profile record number, not the name, is used when specifying a profile access list)


    uconf-driver action=add table=user name=jpublic cs_language=fr
    ct_fullname="John Q. Public" cr_leveltype=3
    ct_password=change%me cs_rlist=!23!

Add a filter to remove hits with the localhost ip (


    uconf-driver action=add table=filter name=localhost-ip cr_filtertype=exclude cs_filterfield=4 ct_filter=127.\0.\0.\1

Extract all profile information for a specific profile

uconf-driver action=get table=profile

Extract a single parameter for a specific profile


    uconf-driver action=get_parameter table=profile parameter=ct_defaultpage

Delete a user


    uconf-driver action=delete table=user name=myuser

Examples of pseudo-code scripts to perform tasks

This section gives examples in pseudo-code, which could be easily translated into a shell script or Perl script depending on the needs of the application.

Apply a German language setting to all users


    $user_count = `uconf-driver action=nrecords table=user`;
    for ($i = 1 ; $i <= $user_count ; $i ++) {
      uconf-driver action=set_parameter table=user entry=$i ct_language=ge;
  • }

Add a new profile/task/logsource/user

Note that in this example, uconf-import takes care of building the proper mappings so that the new profile, task, log source and user will all be properly cross referenced ? e.g. the log source is associated with the profile and vice- versa, the user will be in the profile?s list of authorized users and vice-versa, etc. It should also be noted that the Task is associated to the Profile because they share the same Name tag.

Step 1: Some automated provisioning application creates the following text file (/path/to/text-file):

<Profile Name="">

<Task Name="">

<Logfile Name="">

<User Name="newuser">
ct_fullname="New User"

Step 2: Call the uconf-import utility to import the new profile and user into Urchin:


    uconf-import -f /path/to/text-file

Sample script to add a Profile/Task/LogSource/User using uconf-driver


# Proof-of-concept Bourne shell script for adding a Profile,
# Task, Log Source and User record set to the Urchin configuration.
# The Profile will be set to run at 01:05am daily.

# Define the pertinent information here. Obviously, this should 
really be
# stuff that's parsed from the command line.

cd /path/to/urchin/util

# Add Profile
p_recnum=`./uconf-driver action=add table=profile name=$domain
ct_name=$domain ct_website=http://www.$domain

# Add Task
t_recnum=`./uconf-driver action=add table=task name=$domain
ct_name=$domain cr_frequency=5 cr_enabled=on cs_hour=1 cs_minute=5

# Set proper cross reference from Profile to Task
recnum=`./uconf-driver action=set_parameter recnum=$p_recnum

if [ "$recnum" != "$p_recnum" ]; then
        echo "Failed to associate profile with log source"

# Add Log Source
l_recnum=`./uconf-driver action=add table=logfile name=$domain
cr_action=2 ct_name=$domain cr_type=local ct_loglocation=$logfile
cs_logformat=ncsa cs_rlist=!$p_recnum!`

# Set proper cross reference from Profile to Log Source
recnum=`./uconf-driver action=set_parameter recnum=$p_recnum
if [ "$recnum" != "$p_recnum" ]; then
        echo "Failed to associate profile with log source"

# Add level-3 admin for this Profile
u_recnum=`./uconf-driver action=add table=user name=$username
ct_name=$username ct_password=$password ct_fullname="$domain Admin"
cs_language=$language cs_adminlevel=3 cs_rlist=!$p_recnum!`

# Set proper cross reference from Profile to User
recnum=`./uconf-driver action=set_parameter recnum=$p_recnum

if [ "$recnum" != "$p_recnum" ]; then
        echo "Failed to associate user with profile"