MOM Performance issues, Agent disconnections, unable to login to the workstation

Document ID : KB000045655
Last Modified Date : 14/02/2018
Show Technical Document Details

Symptoms:
-MOM / EM UI is very slow after restart
-Performance is poor and agents therefore disconnect
-logon with Workstation is almost not possible.
-CPU usage is low
-Enterprise Manager responses slow to WebView
-MOM log shows lots of warnings messages:

... loadbalancing (?): [Manager.LoadBalancer] LoadBalancerBean: getCollectorReconnectList(...) failed to determine MOM socket port for socketType=default and # of entries in fMOMSocketInfos=0; using default momPort=

and

... [Manager.IncomingMessageDeliveryTask] Cannot deserialize message while reading from node ...

MOM threaddump shows all Outgoing Delivery threads got stuck on nio writing as below:

...
"Outgoing Delivery 10" prio=10 tid=0x00007fdef800a000 nid=0x19cf runnable [0x00007fdf3776d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000042b2297d0> (a sun.nio.ch.Util$2)
- locked <0x000000042b2297c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000042b2297e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.wily.isengard.postofficehub.link.v1.server.OutgoingMessageDelivererNio.waitForWriteNotification(OutgoingMessageDelivererNio.java:211)
at com.wily.isengard.postofficehub.link.v1.server.OutgoingMessageDelivererNio.run(OutgoingMessageDelivererNio.java:155)
- locked <0x00000003cc5a28a0> (a com.wily.isengard.postofficehub.link.v1.server.OutgoingMessageDelivererNio)
at com.wily.util.concurrent.SetExecutor.doWork(SetExecutor.java:224)
at com.wily.util.concurrent.SetExecutor.access$0(SetExecutor.java:178)
at com.wily.util.concurrent.SetExecutor$WorkerRequest.run(SetExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
...


Workaround:

1. Stop the Enterprise Manager(s)
2. Open the EM_HOME/config/IntroscopeEnterpriseManager.properties, add the below property:

transport.enable.nio=false

If you are using cluster, you need to apply this change in all the EMs (MOM and collectors)

Disabling NIO will switch back to the previous classic socket operations, there is not loss of functionality

3. Start the Enterprise Manager(s)