Processing Historical Logs
Overview
You may wish to process your historical logs after installing Urchin. This is easily accomplished. Simply specify a directory and a partial filename and/or wildcard (including regular expressions) in the Log Manager's Log Settings screens. NOTE: You may not use wildcards on remote HTTP and HTTPS log sources.
How to Process Historical Logs
First, add a Log Source to the system. Click on the Configuration button at top menu, and then the Urchin Profiles -> Log Manager link. On the main screen, click on the Add button at top-right. On the first screen, select Add Local Log Source, and continue. On the next screen, click Browse, which will bring up the File Browser. Locate the correct directory in the left-side window. The right-side window will display the files in the directory, and the left side will display all other subdirectories of currently opened directory. When you are in the correct directory, enter a partial filename and an asterisk (or other regular expression), and click the Verify button. A window will open which will show you all the matches to your pattern. Click any of the filenames to get information on the file - location, size, last modification date, and file permissions. If the pattern match is correct, click OK, and then OK again in the File Browser window. After that specify the Log Source Name and click Next. Specify the log format from drop-down list and click Finish button. In case of valid Log Source Name (which should be unique) press the Done button and you will find your log source Log Manager list.
Next, if it hasn't been already done, associate this log file with a Profile by clicking the Configuration button at top menu, and then the Urchin Profiles -> Profiles button. Once the association has been completed, click the Run/Schedule button next to the Profile in the main Profiles listing. You can schedule the execution of the Profile, or click the Run Now button for immediate processing.
Urchin does not need any sort of log rotation to avoid data duplication. Urchin is equipped with a log tracking capability that ensures only new hits are processed. However, as mentioned above, logs can quickly consume large volumes of disk space, so it is a good idea to periodically compress and archive log files. Because Urchin never needs to re-read log files once they have been processed, it is perfectly acceptable to delete the log(s) after each processing run. However, many people keep logs for a specified amount of time in case they are needed for some reason, such as if a new Profile is created for that site, and historical analysis is desired.
Recommendations
Log management is not essential from the outset, but as logs grow, it becomes important. It is best to decide on a log management plan when you initially deploy Urchin.