How does the agent clamp mechanism work as disallowed agent are not connecting when the threshold is no longer reached

Document ID : KB000117761
Last Modified Date : 18/10/2018
Show Technical Document Details
Issue:
I have a question about how the Disallowed Agent Connection Clamp mechanism works.   In one environment for example we have the clamp, 

<clamp id="introscope.enterprisemanager.agent.connection.limit">

configured to 100 agents, but I would like to better understand what it means and hot it works when this clamp is reached.  

First of all my understanding is that in this case the first 100 agents get connected, but the 101st will be disallowed.   But what happens now when for example ten of the already connected agents are stopped and get inactive (greyed out) in the investigator. Will the previous 101st agent connect automatically now? Will any newly connected agents be accepted immediately?  I’m asking because the environment where we have a clamp of 100 agents configured is a very dynamic cloud environment with agents spinning up and down frequently and we already hit the clamp a couple of times.   However what I don’t understand is that currently the Number of Agents is something around 85. But even though we are not at the Clamp right now there are 12 disallowed agents.
Environment:
APM 9.x, 10.x
Resolution:
In an EM cluster The MOM will try to re-allocate disallowed agents when it does its load balancing routine which occurs periodically based on the property introscope.enterprisemanager.loadbalancing.interval which by default is every 10 minutes. The MOM knows when the collector is not longer clamped as you can see the following format message in the MOM log

7/27/18 05:29:30.877 AM EDT [INFO] [PO:client_main Mailman 3] [Manager] Collector APMRH6U2B4A@5001 reported Clamp not hit for MaxAgentConnections limit.

This occurs pretty much as soon as an agent disconnects from the collector in question but it does not go through the list of disallowed agents until the next load balancing cycle.

For the standalone EM (the environment in which the reported problem was encountered) there was no such functionality and so has been provided in 10.7 SP2 HF16 and later -