If experiencing cluster slowness, have troubleshooted/fixed these issues -- The Collector running slow messages, reduce the number of traces from beng above 500,000, optimized the EM minimum/maximum heap size. And still the issue persists around Collector performance where at any given time, only one collector is showing slow performance, then the number of Business Transactions could be the culprit.
If the number of your Business Transactions are around 2000 or more, then consider reducing the number of Business Transactions. We have seen that a very large number of Business Transactions can cause the cluster to slow down. Also this can be related to DB performance. Please make sure DB is on a separate machine than MOM and has enough resources for CPU and memory. It's not about how much data the database can hold, but more on how fast a database query can be executed.
In the DEBUG logs of the MOM, there may be many of these messages:.
[DEBUG] [PO:main Mailman 4] [Manager.com.wily.apm.tess.isengard.BizDefnitionBeanHelper] evaluating following number of search filters 2
[DEBUG] [PO:main Mailman 4] [Manager.com.wily.apm.tess.isengard.BizDefnitionBeanHelper] passed agent name filter comp <name of Business Process and name of Business Transaction and Transaction Component name> for agent <name of agent)>
Taking thread dumps may reveal that every thread in PO:main Mailman thread pool is process this same incoming message:
This may match what is in the above DEBUG messages. Since every thread in this thread pool is busy handling this request, all other incoming messages are delayed, causing MOM-Collector communication problem.