I would like to understand why the Enterprise Manager marks agents as clamped when the Metric count is not even close to 5000?
By design, if for example an Agent reports 3000 metrics since the beginning, then Historical Metric Count should be around same range.
Otherwise, the Historical Metric Count could be higher as it also counts the metrics that were reported in the history but not reporting any more.
However, in 10.5.x and 10.7.0, there is a bug (DE348526) that affects the reporting of Historical Metric Count if the below conditions are met:
a) If you are reporting String metrics.
Correct behavior: String metric is not historical thus it should not be considered as part of the Historical Metric Count
b) If there are metrics reported as dead and came back again -- when reintroduced they are incorrectly considered as new.
Correct behavior: Most metrics will never be reported as dead, they just stop reporting. Some categories that report metrics as dead are Docker, OpenShift Monitoring and JMX gathering
One way to check if you have hit this issue is to restart the Agent and see if the Historical Metric Count drops.
- If yes then possibly you have hit the issue.
- If it’s still the same as before restart, then most likely there are some old metrics in the SmartStor. You can use the Smartstor tool to list all metrics and count.
Code change: Do not count String metrics and count historical metric just once.
This fix is available in the below releases:
- HOTFIX 10.5.2 #31 and higher
- HOTFIX 10.7 #1 and higher