Log Formats: Apache and IIS

Overview

It is critical to set up your server software's logging properly in order for Urchin to work correctly
and produce full reporting. This article explains the process for the most common servers,
Apache and Microsoft IIS. For maximum reporting depth, it is important to enable logging
to include Referral and User Agent information. To enable unique visitor reporting, it is
necessary to enable cookies as well.

Procedure

Configuring Apache or Microsoft IIS to properly log for Urchin

---

Apache:

Note: Apache uses an NCSA-type log format. By default, Apache generally logs in "common" log format.
For optimal reporting, Urchin requires the "combined" log format with cookies enabled.

1. Decide if you are going to use cookies and the Urchin Tracking Module (UTM),
   or IP-based visitor tracking. UTM-based tracking is the only way to get true unique visitor reporting.
2. Use "vi" or other text editor to open your httpd.conf file.
3. Locate the LogFormat entry or entries.
4. Comment out the existing LogFormat (if any) by inserting a # in front of any lines
   that do not have them already. This deactivates those lines.
5. Insert one of these lines, depending on whether you wish to use the Urchin Traffic Monitor (UTM)
   or not. Be sure that the line is entered into your configuration file as a single line, without carriage returns:

   Non-UTM (IP or IP+UserAgent tracking)

   LogFormat "%l 135594252 t \"r\" >s b \"{Referer}i\" \"{User-Agent}i\"" special

   Urchin Traffic Monitor (UTM):

   LogFormat "%v 135594252 t \"r\" >s b \"{Referer}i\" \"{User-Agent}i\" \"{Cookie}i\"" special

6. Examine the <VirtualHost> entry you wish to enable this logging for. Insert a # in
   front of any existing TransferLog or CustomLog entries to deactivate it/them,
   and insert the following (editing the pathname appropriately for your environment):

   CustomLog /www/serverlogs/www.urchin.com_cookie_log special

   Note the "special" text at the end of the line. This is the name of this particular CustomLog,
   as specified in the LogFormat entry above. You can name your log format anything you
   like; "special" is a common choice for non-standard formats.

7. Write/quit the edit of httpd.conf.

8. IMPORTANT: run this program to check your syntax:

   apachectl configtest

   You should get syntax ok. If not, check the httpd.conf file and fix any errors.

9. Restart Apache. The preferred method is with this script (included with Apache):

   apachectl restart

10. Check the logging. Open a browser and hit the site in question a few times. Examine the log file
    (as specified in #6 above.) You should see a several hits being written to the log. For standard
    "combined" log format, a log line will look similar to this:

    64.40.51.27 www.urchin.com - [28/Aug/2002:15:11:01 -0700] "GET /images/urchin_header_logo.gif HTTP/1.1" 200 3017 "http://www.urchin.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

    If you have configured the UTM on the site, a log line will look similar to this
    (note the additional cookie field with the embedded UTM information):

    64.40.51.27 www.urchin.com - [28/Aug/2002:15:11:01 - 0700] "GET /images/urchin_header_logo.gif HTTP/1.1" 200 3017 "http://www.urchin.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)" "__utm1= 459278579.1030573053; __utm2=1030573053; __utm3=1030573053"

---

Microsoft Internet Information Server (IIS):

Note: Microsoft IIS uses a W3C logging format.

Urchin can provide very basic reporting if your log files have, at the very least, the following fields:

• Date
• Time
• C-IP
• CS-URI-Stem
• SC-Status
• SC-Bytes

These are required fields. Without them, you will not get meaningful reporting, and any reports produced
may contain erroneous information. However, this minimal logging does not provide enough information
for Referral and Browser reporting. Please refer to the following session to set up logging properly.

Setting IIS logging properties

IIS logging properties are configured either separately for each domain on the server, or globally.
For servers with more than a few domains, the global option is recommended.

The following steps will ensure that the required log file fields are being recorded. If you elect to
log additional fields, Urchin will just ignore them at processing time. However, logging unneeded
fields will increase the size of your log files so it is best to only log the fields needed by Urchin.

1. Open your IIS manager.

   Start -> Settings -> Control Panel -> Administrative Tools -> Internet Services Manager.

2. Expand the server tree to display all domains.
3. Right click and select properties for the domain in question.
4. Select the Web Site tab in the Properties Window.
5. Select Enable Logging and change the Active log format pulldown menu to
   display W3C Extended Log File Format.
6. Click on the Properties button next to the Active Log Format window.
7. Select the Extended Properties tab.
8. Place a check mark next to the Date and Time fields
9. In the Extended Properties area, check the following fields:
   Client IP Address [ c-ip ]
   User Name [ cs-username ]
   Method [ cs-method ]
   URI Stem [ cs-uri-stem ]
   URI Query [ cs-uri-query ]
   Protocol Status [ sc-status ]
   Bytes Sent [ sc-bytes ]
   User Agent [ cs[User-Agent] ]
   Referer [ cs[Referer] ]
   Cookie [ cs[Cookie] ] (This field only required for UTM tracking)
   
10. IMPORTANT: You should disable Process Accounting as it provides no useful information for Urchin and may cause log parsing problems.
11. Select Apply and OK on each window to save your settings. It is not necessary to restart IIS.
12. Your logs will immediately begin logging according to the new settings.