Seeing uncaught exception on MOM in the threadpool.

Document ID : KB000012576
Last Modified Date : 14/02/2018
Show Technical Document Details
Question:

 I am seeing this uncaught exception on MOM in threadpool.  What does it mean?

 

[ERROR] [pool-11-thread-286204] [Manager] Uncaught Exception in Enterprise Manager: In thread pool-11-thread-286204 and the message is com.wily.util.exception.UnexpectedExceptionError: Tranport for the registry service at address: {1} is down

[WARN] [PO:main Mailman 2] [Manager] Unable to send signal for clearing denied agents to collector "1.2.3.4@5001"

com.wily.isengard.messageprimitives.ConnectionException: Tranport for the registry service at address: {1} is down

at com.wily.isengard.postofficehub.ClonedRegistry.getEntry(ClonedRegistry.java:141)

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:317)

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:270)

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.getService(MessageServiceFactory.java:132)

at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadRebalancer.getLoadBalancerAdmin(ClusteredLoadRebalancer.java:65)

at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadBalancer.clearDeniedAgentsOnAllCollectors(ClusteredLoadBalancer.java:704)

at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadRebalancer.rebalance(ClusteredLoadRebalancer.java:275)

at com.wily.introscope.server.beans.loadbalancer.ClusteredLoadRebalancer$RebalanceTask.run(ClusteredLoadRebalancer.java:1095)

at com.wily.EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:728)

at java.lang.Thread.run(Thread.java:745)

Environment:
All APM Releases.
Answer:

Basically the messages mean that MOM has lost the connection to the Collector. This generally happens if the Collector was disconnected or running slow around or before the time that the above exception happened.

This warning message is logged by the MOM when it couldn't deliver the notification to the collector for clearing denied agents upon load rebalancing.  This is usually an indication of the Collector having a connectivity issue with MOM or its message queue is already full.  MOM will try sending the notification periodically on each load balancing cycle.  One thing to check would be the current value of introscope.enterprisemanager.loadbalancing.interval.

 This also could happen every so often if the Collector was restarted and has not yet reconnected to MOM at the time of load rebalancing.  However, if these warning messages are occurring repeatedly, then it would likely be a side-effect from some other connectivity issue or or performance issue.