Urchin 7 Parallel Profile Processing
Overview
Previous versions of Urchin processed profiles in serial manner. For example, if it took 1 minute to process 1 profile, processing 100 profiles would require approximately 100 minutes.
To address this issue, Urchin 7.0 introduces parallel profile processing. Now, several profile processing jobs can be run simultaneously. This functionality targets middle and high volume enterprise clients:
- hosting providers
- clients with huge number of profiles
Configuration of parallel profile processing
By default, a newly installed or upgraded instance of Urchin 7.0 allows only a single profile to be processed at a time. To enable parallel profile processing:
-
Uncomment the option MaxConcurrentProfileTasks in <urchin_home>/etc/urchin.conf and set it to necessary value.
- For example:
MaxConcurrentProfileTasks: 2
- Restart Urchin services:
urchinctl restart
Note: Urchin profile is processed exclusively by one instance of the processing engine.
Restrictions
The profile processing job is extremely RAM- and IO-hungry. Blindly increasing the parameter MaxConcurrentProfileTasks will most likely exhaust system resources and significantly degrade processing speed.
You should increase MaxConcurrentProfileTasks only:
- if you have enough RAM to fit several jobs in memory
-
when profiles and log sources are physically located on different physical hard disk drives or you have a high-speed hard disk drive
- if Urchin is used in Data Center mode
Approximate resource consumption:
|
Number of parallel jobs |
RAM, MB |
IO read, MB/s |
IO write, MB/s |
|---|---|---|---|
|
1 |
1000 |
10 |
5 |
|
2 |
2000 |
20 |
10 |