Overview
Log management is an important concern when running software such as Urchin. Because busy sites will build up large log files fairly quickly (up to several gigabytes in some cases in one month), log management should be considered carefully. It is recommended that a standard log rotation practice be established. Please see the Knowledgebase article on Log Rotation Best Practices for further information on establishing such a procedure.
That said, Urchin 4 does not need any sort of log rotation to avoid data duplication, as it is equipped with a log tracking capability that ensures only new hits are processed when reading a log file that is having data appended to it continually. However, as mentioned above, logs can quickly consume large volumes of disk space, so it is a good idea to periodically compress and archive log files. Because Urchin never needs to re-read log files once they have been processed, it is perfectly acceptable to delete the log(s) after each processing run. However, many people keep logs for a specified amount of time in case they are needed for some reason, such as if a new Profile is created for that site, and historical analysis is desired.
How to Manage Logs
Starting with Urchin 4.1 each Log Source has a Log Destiny setting that has the options Don't Touch, Archive/Compress, and Delete. Once all Profiles that are utilizing a Log Source have finished their processing, Urchin uses the Log Destiny setting to determine the disposition of the Log Source. The Log Destiny setting is accessible under the Advanced Settings tab for a given Log Source.
Recommendations
It is recommended to at least have your Log Destiny setting to Archive/Compress so that you save disk space if you want to keep your logs around for historical reasons. If you are comfortable with the fact that once you've processed a log that it is removed, then you can choose a Log Destiny of Delete. However, realize that this means you will not have the option of rerunning Urchin against that log in the future unless you have a backup elsewhere.
Warnings
- If Log Destiny for a remotely retrieved Log Source is set to Don't Touch, then that log will grow continually unless there is some process external to Urchin that is handling log management. Since Urchin must transfer a copy of the remote logfile to the local system before processing, as the log file grows it will take Urchin longer and longer to transfer the file. This will have the side effect of lengthening your overall Urchin run time.
- Do not use the Archive or Delete options with a Log Source if you are processing live logs! This will cause a loss of log data.
See Also