MOM not connecting to collectors - Invalid control byte: 0

Document ID : KB000123120
Last Modified Date : 18/12/2018
Show Technical Document Details
Issue:
We have a cluster where the MOM is failing to connector the collectors and reporting a lot of warnings in the MOM log

12/18/18 04:57:42.524 AM EST [WARN] [UnknownHub Hub Receive 461] [Manager.Cluster] Invalid control byte: 0

In addition we are also seeing the following messages in the collector logs

12/18/18 03:15:18.705 AM EST [INFO] [Thread-ClusterTopologyPoller] [Manager.AppMap] Cannot send EM topology due: 'Invalid Registry Path Exception' Will retry.

 
Environment:
Windows 2008 APM 10.7
Cause:
The environment had been created from clones of existing servers and the configuration details modified to suit the new envirionment. However the collectors, DB and MOM were not all at the same patch level. MOM and APM DB were at 10.7 SP2 (build 115) and the collectors were at 10.7 GA (build 45)
Resolution:
Applying 10.7 SP2 to the collectors resolved the problems
Additional Information:
enabling DEBUG logging in the MOM log reveals following types of messages - for clarification of problem.

12/18/18 04:57:42.524 AM EST [WARN] [UnknownHub Hub Receive 461] [Manager.Cluster] Invalid control byte: 0
12/18/18 04:57:42.525 AM EST [DEBUG] [UnknownHub Hub Receive 461] [Manager.Cluster] Corrupted stream while reading from node at: Socket Transport connected with Socket[addr=/xxx.xxx.xxx.xxx,port=5002,localport=58999]
java.io.StreamCorruptedException: Invalid control byte: 0
    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:282)
    at com.wily.introscope.spec.server.beans.clusters.ConnectedCollector.<init>(ConnectedCollector.java:155)
    at sun.reflect.GeneratedConstructorAccessor523.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:717)
    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:346)
    at com.wily.isengard.messageprimitives.MethodResultMessage.readExternal(MethodResultMessage.java:65)
    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readIsengardDataObject(IsengardObjectInputStream.java:824)
    at com.wily.isengard.postofficehub.link.v1.IsengardObjectInputStream.readObject(IsengardObjectInputStream.java:346)
    at com.wily.isengard.postofficehub.link.v1.IncomingMessageDeliveryTask.deliverNextMessage(IncomingMessageDeliveryTask.java:80)
    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.receiveIncomingMessages(IncomingRouteConnector.java:170)
    at com.wily.isengard.postofficehub.link.v1.IncomingRouteConnector.doTask(IncomingRouteConnector.java:89)
    at com.wily.isengard.util.thread.AThreadedExecutable.run(AThreadedExecutable.java:192)
    at java.lang.Thread.run(Thread.java:745)