Optimizing Policy Engine Performance

Document ID : KB000101864
Last Modified Date : 15/06/2018
Show Technical Document Details
Introduction:
Policy engines can process events such as emails or files arriving from an external source and apply policy triggers to these events. Typically, events are allocated to individual policy engines by a policy engine hub. Policy engines enable CA Data Protection to integrate directly with :
  • Email servers.
  • Archive solutions.
  • Scanned items.
  • Imported files, emails, and IM conversations.

 
Question:
As policy engines have a wide range of input sources and volumes can be high, is there an optimum performance configuration or anything that should be considered to optimize performance?
 
Environment:
CA Data Protection 14.x\15.x
Answer:
Physical verses Virtual.
There is a drive towards virtualization as this can reduce the total cost of ownership (TCO) by making it quicker and easier to commission machines, however the CA Data Protection Policy Engines are designed to enable maximum use of the physical CPU and that performance could be constrained in a virtual environment.  To clarify, CA Data protection Policy Engines do run on virtual platofrms, but performance is improved by having a dedicated CPU rather than running on a VM host where Data Protection PEs only get a slice of the core and share it with other processes.

Maximum Number of Concurrent Operations
The "Maximum Number of Concurrent Operations" machine policy setting can be configured to define the maximum number of emails that can be processed simultaneously.  This defaults to 5. and it enables the policy engine to make the most efficient use of system resources. 

You may want to increase the maximum limit if the policy engine is running on a multiprocessor computer.
If this maximum limit is reached, the policy engine delays accepting any further emails from the policy engine hub until the number of emails being processed falls below this maximum limit. This means that when an email completes processing, another is accepted, so maintaining the number of emails at the maximum limit. For example, if five emails finish processing simultaneously, the policy engine immediately accepts five new emails.

Example;
If your system has at least 1x 16-core double-threaded chip and that the core(s) is(are) fully dedicated to the PE function (ie this is not a VM host on which the PE is running as a guest) you could set the conncurrency value to 32.  

If you had, for example an Intel Xeon E7-4850-v4 processor which has 16 cores and 32 threads, you could set the concurrency to a value of 32 for PE concurrency and this will create 32 processing threads on the PE. However, because this processor has two threads per core, it may be possible to set the concurrency to 48.  Higher setting are possible but we would recommend exhaustive testing to see if there is any significant performance improvement or instability. 

Memory.
In addition to the number of physical cores, the other limitation is the amount of Memory.  Each PE Processing thread requires an amount of RAM.  If we were to assume each thread requires 1MB per thread, you would need at least 32 GB of Memory to support the 32 threads concurrently.

Note: The figures above are only illustrative and represnt virtual memory and not necessarily physical.  This means that the SWAP / PAGE file should be located on fast I/O storage. 


Policy Engine Scaling
While planning your PE deployment you should consider scaling and IO performance. 

As far as the storage on the PE is concerned you need to consider the follwoing things: 
1. Is the PE using a local SQL database instance? If it is, the storage should be configured as per the client’s SQL Server Storage configuration standards / best practice. This typically means that the PE will have at least 4 volumes – each on dedicated spindles (not separate partitions on the same disk).  For example: 
  • C: drive (OS/SWAP); 
  • D: drive (apps/binaries/SQL DB MDF); 
  • E: drive (SQL LOGS LDF/ BAK); 
  • F: drive (Data Protection Data / PE-User TMP directory); 

This will yield optimal I/O configuration which should translate into higher scale in Msg/Sec processing. 

2.  The PE caches messages on the disk to the PEService Account user’s TMP directory – which is typically on the C drive. Moving the PE Service Account’s User’s TMP directory to faster storage (read storage with higher IOPS) will ultimately produce a better scaling and better performing PE in terms of  Msg/Sec processing. 

Additionally, the average message size plays into the formula. Each Policy Engine (PE) thread is capable of processing x-MB per second which (conservative estimates place this at 2MB/sec).  As long as the average message size is less than 2MB/sec, then each PE should be able to process at least 32 msgs/sec concurrently, however typically it is much higher than that because the avg msg size is closer to 100-200KB. 


All of the above facets will affect a PE’s processing latency (this can be monitored via a performance counter). As long as the processing latency is less than 200 ms, a dedicated 32-thread PE, configured as above, with an assumed avg msg size of 100KB-150KB, should yield upwards of 120 Msgs/Sec. 

 
Additional Information:
Optimizing PE performance needs to consider a large number of variables specific to each customer environment.  While this knowledge document offers some areas for consideration, however refining performance may require testing in a controlled environment.  CA Services can assist with Policy Engine refinement and testing as a billable service.  To engage with CA Services please reach out to your CA Account Manager.


Note:
CA DataMinder, CA DLP, Orchestria, Orchestria APM and Orchestria EPM are all previous brands for the product now known as CA Data Protection.