Urchin 7 Parallel Profile Processing

Urchin 7 Parallel Profile Processing

Overview

Previous versions of Urchin processed profiles in serial manner. For example, if it took 1 minute to process 1 profile, processing 100 profiles would require approximately 100 minutes.

To address this issue, Urchin 7.0 introduces parallel profile processing. Now, several profile processing jobs can be run simultaneously. This functionality targets middle and high volume enterprise clients:

  • hosting providers
  • clients with huge number of profiles

Configuration of parallel profile processing

By default, a newly installed or upgraded instance of Urchin 7.0 allows only a single profile to be processed at a time. To enable parallel profile processing:

  1. Uncomment the option MaxConcurrentProfileTasks in <urchin_home>/etc/urchin.conf and set it to necessary value.

    • For example:

 

MaxConcurrentProfileTasks: 2
  1. Restart Urchin services:
p class="line867">
urchinctl restart

Note: Urchin profile is processed exclusively by one instance of the processing engine.

Restrictions

The profile processing job is extremely RAM- and IO-hungry. Blindly increasing the parameter MaxConcurrentProfileTasks will most likely exhaust system resources and significantly degrade processing speed.

You should increase MaxConcurrentProfileTasks only:

  • if you have enough RAM to fit several jobs in memory
  • when profiles and log sources are physically located on different physical hard disk drives or you have a high-speed hard disk drive

  • if Urchin is used in Data Center mode

Approximate resource consumption:

Number of parallel jobs

RAM, MB

IO read, MB/s

IO write, MB/s

1

1000

10

5

2

2000

20

10