Custom Data Maps

Custom Data Maps

Data maps describe set of tables, used as storage for data calculated from log files. Predefined tables are as follows:

• T - totals;
• TR - transactions;
• H - histogram;
• V - visitors;
• 1-n - predefined aggregate tables described in the data map file.

The default set of tables is stored in the <Urchin>/lib/reporting/profiletypes/Default/datamap.dm2 file. It uses the following format for tables:

• Each table entry uses one row and should contain the following fields: TABLE AFIELD BFIELD CFIELD DFIELD EFIELD REQUIREMENTS SPECIAL IFIELDS PPFIELDS;
• Columns should not contain spaces;
• Columns are separated by one or more spaces;
• If the value is empty it should be replaced by the '-' sign;
• TABLE - unique identifier of table, maximal value is 200;

• AFIELD-EFIELD - string fields from <Urchin>/lib/reporting/logformats/fieldlist.txt;

• REQUIREMENTS describes which fields (A-E) must be present in hit for latter to get into table, kind of primary key constraint.

e.g. given that A is required in table T1, A refers to "IP-address" (cfield=4), and hit doesn't contain IP-address, as a result whole hit will be bypassed and won't get into T1. however, if another table T2 refers to "IP-address", but REQUIREMENT contains no A, hit will get to T2.

• SPECIAL - used to link records from aggregates table to records from internal tables (sessions, visitors, transaction, transaction items, paths), kind of foreign key. E.g. table #1 from default datamap file links source, medium and campaign to session (with "source" as primary key), table #2 links source, medum, term and content to session (with "source" as primary key), table #12 links request stem and page title to paths. It is not recommended to make any changes in tables with special flags or somehow use flags in custom tables, since it may lead to clobbering of foreign keys.
  • S1-S11 - session link
  • P1 - path link
  • PARAM - special parameter
  • T1-T2 - transaction link
  • I0 - I1 - item link
  • IPATH - initial path
  • GTHRU - goal click thru
  • GPATH - reverse goal path
  • CTHRU - click thru
• IFIELDS - metrics. Tables store preprocessed set of metrics to optimize data extraction speed. i.e. when engine processes logs, it makes all required calculations for each unique combination of dimensions (or cfields) to avoid this phase during extraction. E.g. given cfields: IP-address, ifields: visits. during processing new table is created to store precalculated visits for each IP-address. report, that is built on top of table, extracts the data right away almost without any extra calculation effort. Valid values for IFIELDS column are described below;
• PPFIELDS - post processing fields (from range of ifields), the data for the post processed fields is determined from the session record and its associated path, transaction, and item records. Using ppfields in a custom report is not recommended.

To create custom data map entry, follow the above instructions to create a new row. To avoid breaking the data map, make sure that each new table entry uses a unique number.

Example of row in data map file:

TABLE	AFIELD	BFIELD	CFIELD	DFIELD	EFIELD	REQUIREMENTS	SPECIAL	IFIELDS	PPFIELDS
1	139	138	137	-	-	A	S1	5,11,15,19,24,25,26,27,28,29,30,31	-
Table number	Field A from the record to be used	Field B from the record to be used	Field C from the record to be used	Field D is not used	Field E is not used	Only A field must contain data	Session link 1	Calculated fields	No post processing fields

Calculated fields used for IFIELDS column:

1. Hits
2. Validhits
3. Errorhits
4. Bytes
5. Pages
6. Nonpages
7. Entrancepages
8. Exitpages
9. Bouncepages
10. Pagetime
11. Visits
12. Visitors
13. Newvisitors
14. Priorvisitors
15. Transactions
16. Customers
17. Newcustomers
18. Priorcustomers
19. Revenue
20. Tax
21. Shipping
22. Items
23. Itemrevenue
24. Responses
25. Impressions
26. Clicks
27. Cost
28. Goals1
29. Goals2
30. Goals3
31. Goals4
32. Goalstarts1
33. Goalstarts2
34. Goalstarts3
35. Goalstarts4
36. Score