u3importer: Urchin 3 Data Import Utility

Overview

The u3importer is a command line utility found in the util directory of the Urchin distribution. This utility allows for the importing of report configurations from an existing Urchin 3 config file as well as the data associated with each Urchin 3 report.

The u3importer has two modes; an interactive mode which prompts the user for the required information to perform the import operations, and a non-interactive mode that requires no dialog (more suited to unattended scripting operations).

When run in interactive mode without any command line arguments, for each <Report> block and its associated directives in the Urchin 3 config file the utility will:

  1. Create an Urchin 5 Profile, Log Source, and Task
  2. Create any associated Filter entries
  3. Read the Urchin 3 databases and write the data into new Urchin 5 databases in the appropriate location in the Urchin 5 distribution

In non-interactive mode using the "-c" option (explained below), u3importer will simply take Urchin 3 databases and import them into new Urchin 5 database files without making any changes to the Urchin 5 configuration.

Important Note: The utility must be run on the same operating system platform that originally created the Urchin 3 report data. It is unable to read and convert data created on a different operating system platform and attempting to do may cause unexpected termination of the utility. If you wish to upgrade from an Urchin 3 installation on one operating system to a new Urchin 5 installation on another platform type, you should first install a temporary copy of Urchin 5 on the old platform. Next, run the u3importer to create an interim Urchin 5 configuration and profile data. Since the resulting Urchin 5 profile data is platform-independent, this data can then be moved over to your permanent Urchin 5 installation on the new platform. Please note that it is not necessary to license the Urchin 5 distribution on your old platform in order to use the u3importer utility.

Please be sure to read the Limitations section below for a more detailed explanation of the limitations of importing Urchin 3 data.

Usage

Usage of the utility is as follows:

  u3importer [-v] (prints usage and exits)
  u3importer (interactive mode)
  u3importer [-c urchin3-report-data-path urchin5-report-data-path]
     (non-interactive mode)

Data Importing Procedure

When upgrading from Urchin 3, u3importer should be run before creating any new Urchin 5 profiles, if possible, since it must create a new profile for each Urchin 3 report it reads. The utility will not add data to an existing profile. Instead if a profile of the same name as it is trying to import exists, it will create a similar name with the number 2 appended to it and import the data into that profile. Therefore, it is strongly advised to run u3importer before normal operation begins under Urchin 5. It is also recommended to disable the automation of any existing Urchin 3 processing, so that new log files are not discarded or lost during the upgrade process. This will also ensure that the Urchin 3 data is not changing while running u3importer to import your databases.

Interactive Mode Operation

UNIX

Inside a command line shell on the system where Urchin is installed. Change directory to the Urchin util directory, and execute u3importer like so:

    ./u3importer

Windows

On the system where Urchin is installed, open a Command Shell by going to Start->Run..., enter "cmd" and hit Enter. Once the shell window launches, type:

  C:
  cd \Program Files\Urchin\util
  u3importer.exe

Step 1: Locate the Urchin 3 configuration
The u3importer utility will prompt for the location of the Urchin 3 configuration file. A suggested location is provided. To accept the suggestion, simply press return. Otherwise, enter the complete path to the config file located in the Urchin 3 folder. Wherever Urchin 3 was installed, there should be a config file located in that Urchin 3 folder. On Unix systems, this could be /usr/local/urchin3/config. On Windows systems, this could be C:\Program Files\Urchin3\config. If you cannot find the Urchin3 installation, please contact your system administrator for details.

Step 2: Import Urchin 3 configuration profiles
Once the utility locates the Urchin 3 configuration, it will list all of the sites that exist in the configuration and prompt you for which ones to import. To import all profiles, press enter. To import only select profiles, type Y or N as each profile is prompted. Before continuing to the next step, you may verify that the configurations were imported correctly by inspecting the Urchin 4 Configuration interface.

Step 3: Import Urchin 3 data
After importing the configurations, the utility will then prompt to import the data associated with each profile. Importing the data will allow you to view Urchin 3.x historical reports under the new Urchin interface. To import data for all profiles, press enter. To import data for only select profiles, type Y or N as each profile is prompted.

Non-interactive Mode Operation

In non-interactive mode u3importer simply reads existing Urchin 3 databases and creates the associated new Urchin 5 databases. It is up to the user to make sure the Urchin 5 databases are located properly within the Urchin distribution and that the necessary Profiles, Log Sources, and Tasks are created to fully configure the site. To launch u3importer in non-interactive mode without additional dialog, use the "-c" option like so:

  u3importer -c /path/to/urchin3_udata
/path/to/urchin5_databases
The path to your Urchin 3 data should point to the same directory path as shown in your Urchin 3 config file ReportDirectory directive for a given site. The path to the place to create your Urchin 5 directory can be anywhere. But if you want it to automatically become a part of your Urchin 5 configuration when you later add a profile for a site, you should make the path point to the data/reports subdirectory of your Urchin distribution. As an example, if you have a site named test.urchin.com in your Urchin 3 configuration, then in the report block for that site will be ReportDirectory directive similar to:
  ReportDirectory: /urchin3/test.urchin.com/
Assuming your Urchin 5 installation is in the default location of /usr/local/urchin5, then to convert the Urchin 3 databases for this site and have them put in the proper location in your Urchin 4 installation, you would run the following command (line breaks added for readability, this command would be invoked on a single line):
  u3importer -c \
    /urchin3/test.urchin.com \
    /usr/local/urchin5/data/reports/test.urchin.com
In a scripted environment, after running this command you would typically use the uconf-import or uconf-driver utilities to create the associated Profile, Log Source, and Task configuration records to complete your migration for this site.

Limitations

Urchin 3 provided only a subset of the reports that are available by default in Urchin 5, and in many cases the reporting data was maintained only on a monthly basis. Therefore, not all Urchin 5 reports will be populated when importing Urchin 3 data. The following outlines the limitations of importing data from Urchin 3:

  1. Unpopulated reports: the following reports were not present in Urchin 3, so they will be blank in Urchin 5 after importing Urchin 3 data.
    1. Traffic -> (hourly view for all graph reports)
    2. Visitors & Sessions (all reports)
    3. Pages & Files -> Downloads
    4. Pages & Files -> All Files
    5. Pages & Files -> Directory by Files Drilldown
    6. Pages & Files -> Page Query Terms
    7. Navigation -> Click Paths
    8. Navigation -> Length of Pageview
    9. Referrals -> Referral Errors
    10. Domains & Users -> IP Addresses
    11. Domains & Users -> IP Drilldown
    12. Domains & Users -> Usernames by Bytes
    13. Browsers & Robots -> Browsers by Bytes Drilldown
    14. Browsers & Robots -> Platforms by Bytes Drilldown
    15. Browsers & Robots -> Robots by Hits Drilldown
    16. Browsers & Robots -> Robots by Bytes Drilldown
    17. Client Parameters (all reports)
  2. Navigation data in Urchin 3 is stored on a monthly basis, whereas in Urchin 5 it is stored daily. When importing Urchin 3 data, the data is all consolidated into the first of the month for the following reports:
    1. Navigation -> Entrance Pages
    2. Navigation -> Exit Pages
  3. No imported Profiles are scheduled to run, since there was no concept of scheduled tasks in Urchin 3. Use the uconf-schedule utility to globally schedule all imported Profiles to run at a certain time.
  4. Unique Log Sources are created for every TransferLog entry in the Urchin 3 config file, even if muliple Urchin 3 reports use that same TransferLog. You may want to consolidate the Log Sources after importing your Urchin 3 data if multiple Profiles share the same log file.
  5. Unique Filters are created for each FilterIn, FilterOut, and DynamicURL directive encountered in the Urchin 3 config file. Again, you may want to consolidate the imported Filters if you have multiple Profiles or Log Sources that use the same filter.
  6. The u3importer does not perform any type of disk space checking before running, so you will want to ensure that you have enough disk space to perform the import operation. Generally speaking, you will need to allocate at least as much additional disk space as what is currently required for all your Urchin 3 report directories.

Considerations

  1. No checking is done to see if previous data exists in the Urchin databases for the time range covered by the imported data. Therefore, if you run the u3importer tool again, you will double your statistics for that period.
  2. Care should be taken when using the u3importer tool to import data from months that have already been populated with native Urchin 5 data. While the importing process does not overwrite existing data, it does no checking to see if data already exists for a given day and will happily add (possibly duplicate) data to the databases.