z/OS Connect J9 VM dumping where monitored by Introscope

Document ID : KB000121602
Last Modified Date : 28/11/2018
Show Technical Document Details
Issue:
Running the WebSphere agent  for monitoring : z/OS Connect 03.00.11/wlp-1.0.21.cl180220180619-0403 IBM J9 VM version pmz6480sr4fp10-20170727_01 SR4 FP10
The agent starts, but after 20 seconds, z/OS Connect produces dumps

Examples from ffdc log

Stack Dump = java.lang.VerifyError: JVMVRFY041 invokespecial of wrong initializer; class=com/ibm/ws/kernel/launch/internal/ServerCommandListener, method=init(ILjava/io/File;)V, pc=620
Exception Details:
  Location:
    com/ibm/ws/kernel/launch/internal/ServerCommandListener.init(ILjava/io/File;)V @620: JBinvokespecial
  Reason:
    Type 'com/wily/introscope/agent/probe/io/ManagedFileOutputStream'

Stack Dump = java.lang.VerifyError: JVMVRFY041 invokespecial of wrong initializer; class=com/ibm/ws/kernel/launch/internal/ServerCommandListener, method=asyncResponse(Ljava/lang/String;Ljava/nio/channels/SocketChannel;)V, pc=103
Exception Details:
  Location:
    com/ibm/ws/kernel/launch/internal/ServerCommandListener.asyncResponse(Ljava/lang/String;Ljava/nio/channels/SocketChannel;)V @103: JBinvokespecial
  Reason:
    Type 'com/wily/introscope/agent/probe/lang/ManagedThread'
Environment:
Agent release 10.5.2
Cause:
Examples from ffdc log

The logs were showing evidence of features of full instrumentation

com/wily/introscope/agent/probe/io/ManagedFileOutputStream
com/wily/introscope/agent/probe/lang/ManagedThread

These correspond to these tracer groups that are active in full instrumentation level in toggles-full.pbd

TurnOn: FileSystemTracing

TurnOn: ThreadTracing

We can attribute this to the underlying architecture in USS being still mainframe and not distributed.
Resolution:
The recommendation would be to run with typical instrumentation, and make sure all socket-related instrumentation is commented out, this includes the following options to check in toggles-typical.pbd:

#TurnOn: SocketTracing
#TurnOn: ManagedSocketTracing
#TurnOn: UDPTracing

#TurnOn: NIOSocketTracing
#TurnOn: NIOSocketSummaryTracing
#TurnOn: NIOSelectorTracing
#TurnOn: NIODatagramTracing
#TurnOn: NIODatagramSummaryTracing
Additional Information:
While this is not full WebSphere Application Server, the Agent.jar configuration can be used but AgentNoRedefNoRetrans.jar and NoRedef.profile should be considered if there are further issues with instability