Unable to start the appserver - java.lang.VerifyError: JVMVRFY012 stack shape inconsistent;

Document ID : KB000006668
Last Modified Date : 16/03/2018
Show Technical Document Details
Issue:

Unable to start the appserver, the below exception is reported in the application server log: "java.lang.VerifyError: JVMVRFY012 stack shape inconsistent"
Example 1:
[05/04/17 07:33:24:553 BST] 00000001 ContainerHelp E   WSVR0501E: Error creating component com.ibm.ws.ssl.core.SSLComponentImpl
java.lang.VerifyError: JVMVRFY012 stack shape inconsistent; class=com/ibm/ws/ssl/config/ManagementScopeData, method=<init>(Ljava/lang/String;)V, pc=22
at com.ibm.ws.ssl.config.ManagementScopeManager.currentScopeContained(ManagementScopeManager.java:324)

Example 2:
[INFO    ] FFDC1015I: An FFDC Incident has been created: "javax.resource.ResourceException: java.lang.VerifyError: JVMVRFY012 stack shape inconsistent; class=com/ibm/msg/client/wmq/internal/WMQConnection, method=<init>(Lcom/ibm/mq/jmqi/JmqiEnvironment;Lcom/ibm/mq/jmqi/JmqiMQ;ILcom/ibm/msg/client/jms/JmsPropertyContext;)V, pc=22 com.ibm.ws.ejbcontainer.mdb.internal.MDBRuntimeImpl 782" at ffdc_17.05.03_11.16.

Environment:
All supported APM releases
Cause:

This issue is related to defect DE275471 - US309901 : constructor instrumentation broken with latest Oracle and IBM JVM 1.8 SR4 (8.0.4.0), 7 R1 SR4 (7.1.4.0), 7 SR10 FP0 (7.0.10.0)
CA APM Development worked with IBM Support to resolve the issue : PMR# 36599,756
The response was that the full instrumentation of constructors is discouraged.
This problem with ctor instrumentation will affect the instanceCounter you should not miss anything except InstanceCounts metrics.

Resolution:

As a result of the response to the IBM PMR a new value, "Partial" has been created for the introscope.autoprobe.constructor.policy property. This property works in conjunction with the property introscope.autoprobe.constructor.instrumentation.affected.vendors

1. Stop the Application Server
2. Open the AGENT_HOME/core/config/IntroscopeAgent.profile, set:
introscope.autoprobe.constructor.policy=Partial
introscope.autoprobe.constructor.instrumentation.affected.vendors=Oracle,SAP,IBM
3. Start the Application Server

Please note that not all VerifyErrors are the same, they have different root causes. 
If you get VerifyError with Partial constructor instrumentation, then it’s not the same problem that we worked with IBM but a new one we should investigate, capture a dump of a problem class with following jvm arguments:
-Dintroscope.agent.instrumentation.verification.debug=jdk/nashorn/internal/runtime/ScriptEnvironment -Dintroscope.agent.instrumentation.debugdump.path=<valid and existing path on filesystem>
It will save two copies of class bytes, one before and one after the instrumentation. 
Open a support case and provide the output, this will help CA APM Development team to quickly identify what type of verify error it is from and fix it
In the meantime, you can workaround the issue by disabling the instrumentation of constructors as below:
1. Stop the Application Server
2. Open the AGENT_HOME/core/config/IntroscopeAgent.profile, set
introscope.autoprobe.constructor.policy=None 
3. Start the Application Server
 

Note: the above is true for releases up to 10.5.2.15.
For 10.5.2.16 and later (in effect 10.5.2.SP2) the property has the following default value:
introscope.autoprobe.constructor.policy=Partial
The property introscope.autoprobe.constructor.instrumentation.affected.vendors is deprecated.
The 10.5 documentation now describes this latest behaviour.
For the previous behaviour refer to the 10.3 documentaton

Additional Information:

https://docops.ca.com/ca-apm/10-5/en/implementing-agents/java-agent/java-agent-properties#JavaAgentProperties-AutoProbeProperties
https://docops.ca.com/ca-apm/10-3/en/implementing-agents/java-agent/java-agent-properties#JavaAgentProperties-AutoProbeProperties