Fluctuating agent inventory causes poor Engine Collect Task performance.

Document ID : KB000074323
Last Modified Date : 20/03/2018
Show Technical Document Details
Issue:

Since upgrading ITCM agents to version 12.9 GA, 14.0 GA or 14.0 SP1, the Engines on the Domain Manager that are processing Collect Tasks seem to take longer or perform poorly compared to previous experiences with the product.

Also, when checking inventory reported by the agent, sometimes "Operating System --> System Updates" are reported, and sometimes they are not, as seen by the following comparison between a working agent, and a non-working agent:

User-added imageUser-added image

Environment:
Client Automation (ITCM) -- r12.9 GA, r14.0 GA and r14.0 SP1 only.
Note: This problem was addressed in the r14 SP2 release, therefore it is not affected.
Cause:

The root cause of the problem is with the advanced inventory scanner module (amAdvInvNT.exe) from the AM agent.  In order to inventory a list of operating system updates on Windows, the module parses output from a WMI query.  Due to a preset timeout in the code, sometimes the WMI query would return within the length of time the inventory module would expect, and other times it would not.

As the AM agent works using a delta inventory (difference inventory) from the previous inventory, the AM agent would in one instance create a huge delta, to re-report all the operating system updates, then in the next instance erase all the operating system updates.  Hence the AM agent inventory is observed to be fluctuating.

As a result of this, the Engine on the Domain Manager that is collecting these inventory files, is busy writing, erasing and rewriting this inventory, each time the AM agent changes it, overall contributing to poor collect task performance.  It doesn't help that the System Update inventory is typically the single largest section of inventory reported by the AM agent.

Note: The AM agent reporting of operating system updates should not be confused with the software inventory scanner module (amsoftscan.exe & amswsigscan.exe) of installed software, including Windows patches, nor Software Delivery installation records reported by the SD agent.  These records are not affected by this problem.

Resolution:

To permanently resolve the problem, apply the appropriate patch for your version of ITCM:

r14.0 SP1- T5IZ373 (Configurable SystemUpdates timeo)
r14.0 GA- T5IZ342 (System update inv collect timeou)
r12.9 GA- T5IZ292 (Inventory collect timeout on age)

The patch is applicable to all ITCM agents. 

I would recommend scaling the delivery of this patch to agents in your environment, as once installed, it will cause agents that were not reporting system update inventory, to suddenly resume reporting this inventory, which will cause a bottleneck delay at the collect tasks in the near term.  In the longer term, once agents are patched and are no longer fluctuating this data, the collect tasks will overall have less information to process into the database.

If you have not already, I also highly recommend you edit the schedule for the "Inventory Configuration", on both the Enterprise Manager and all Domain Managers, to limit the hardware inventory scanner to running only once per day.  By default the hardware inventory scanner is unrestricted by schedule, causing the agents to report multiple delta inventories per day, based on events triggers that launch the AM agent.

User-added image
User-added image

Additional Information:
As a published solution is currently not available on the support site, you are encouraged to open a support ticket to request these patches directly from CA Support.  Also available are ready-made software packages, based on WinOffline, that you can drag & drop into your software library, in order to simplify patching agents in the environment.