Using Date Substitution in Log Source Filenames

Overview

In addition to supporting wildcards in log source specifications, Urchin also allows for a special YYMMDD syntax to be used to match a logfile that has a datestamp associated with it, as would typically be created by a daily log rotation mechanism.

Procedure

When creating or editing a Log Source, a YYYYMMDD syntax can be specified within filename portion of the Log File Path field. At log processing time, Urchin will replace the following patterns (all upper case) as specified below:

DD is replaced by the 2-digit numeric date of yesterday, e.g. 01-31
MM is replaced by the 2-digit numeric month of yesterday, e.g. 01-12
YY is replaced by the 2-digit numeric year of yesterday, e.g. 01-99
YYYY is replaced by the 2-digit numeric year of yesterday, e.g. 0001-9999

As an example, a typical daily Apache webserver log rotation scheme creates a log with the datestamp indicating the date of the log entries, e.g. at 1 minute after midnight on 07/16/2002 the log rotation mechanism archives the log:

/var/log/httpd/access.log and saves it as

/var/log/httpd/access.log.20020715 To match this pattern in the log source for an Urchin Profile, you'd simply specify

/var/log/httpd/access.log.YYYYMMDD in the Log File Path and Urchin will automatically look for the previous day's log when it runs that day.

As another example, when Microsoft's IIS webserver is configured to rotate logs daily, it will name the logfile and include the current date as part of the filename, e.g. ex021127.log. Therefore, to process a daily IIS log, you would use a logfile specification something like:

C:\WINNT\System32\LogFiles\W3SVC1 \exYYMMDD.log in the Log File Path field of the Log Source for the Profile.

Please note that the DD, MM, YY and YYYY specifications may appear anywhere in the filename for a log specification, so the following are all valid usage of the pattern matching:

Using YYYYMMDD syntax when logs are rotated multiple times daily

To allow Urchin to process logs that are rotated on more frequently than just a daily basis, you can use a combination of the YYYYMMDD syntax and wildcards to match all logfiles created the previous day. To do this, you would need to ensure that the rotated log file was named consistently, e.g. with an hour appended to the filename. In the Log File Path specification, you'd then use a pattern such as:

/var/log/httpd/access.log.YYYYMMDD* or

C:\WINNT\System32\LogFiles\W3SVC1 \exYYMMDD*.log At log processing times, Urchin will then process all logs matching yesterday's date pattern, with any suffix. As with any use of wildcards in the Log File Path field specification, it is important that Log Tracking for the Profile be enabled to ensure that Urchin does not re-process logs.

Considerations

To determine the date for the replacement pattern, Urchin subtracts 24 hours from the current time, based on the local time. It will properly handle month and year boundaries.

See Also

Log Rotation Best Practices