Modifications to thresholds defined in apm-events-thresholds-config.xml are not always recognized on Enterprise Manager restart.

Document ID : KB000028836
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

Sometimes on startup, not all parameters defined in the apm-events-thresholds-config.xml are read and the default values are applied. 

Restarting the EM may or may not result in a correct reading of all parameters. The processing of this file is not consistent.

The problem is more likely to occur if the EM is running as a Windows service but can also occur when running as a console process.

It is not always clear that there is a problem as no warning message or alert is produced at startup. It may only become evident if one of the modified values is exceeded e.g. live and historical metric clamp values. 

If the problem has occurred, Inspection of the EM log file reveals one of the following entries is missing after startup . Note that the values will vary depending on the settings in the apm-events-thresholds-config.xml file

[Clamp : introscope.enterprisemanager.agent.metrics.limit, Value : 120,000]
[Clamp : introscope.enterprisemanager.metrics.live.limit, Value : 5,000,000]
[Clamp : introscope.enterprisemanager.metrics.historical.limit, Value : 12,000,000]
[Clamp : introscope.enterprisemanager.agent.connection.limit, Value : 800]
[Clamp : introscope.enterprisemanager.disconnected.historical.agent.limit, Value : 400]
[Clamp : introscope.enterprisemanager.events.limit, Value : 1,250]
[Clamp : introscope.enterprisemanager.agent.trace.limit, Value : 1,000]
[Clamp : introscope.enterprisemanager.agent.error.limit, Value : 10]
[ClampType : introscope.clw.max.users, Value : 500]
[ClampType : introscope.workstation.max.users, Value : 40]
[ClampType : introscope.enterprisemanager.collector.cdv.max, Value : 10]

Solution:

If the problem has occurred, a workaround is to hot change the property value that is not correctly loaded. (Only be concerned about those that have been modified from the default value.

You can either edit the file and change the value as shown below:


clamp id="introscope.enterprisemanager.metrics.historical.limit"> <description> Per EM limit. Takes into account metrics with Smartstor data (i.e. live and historical metrics) </description> <threshold value="12000000"/> </clamp>

and set    <threshold value="12000001"/>  

Or have a copy of the file with the non default values set to a slightly different value such as above and then overwrite the exiting file.

The EM will then detect the hot configuration change and implement the missing value(s). If the system is likely to experience this problem, then automate the process of overwriting the apm-events-thresholds-config.xml a short time after startup. 

This will reduce the risk of experiencing problems due to incorrectly set thresholds.

This problem has so far only been seen in APM version 9.5.2 and 9.5.3 and a fix is being produced for a later version.