APM Introscope Java Agent causing High CPU Usage

Document ID : KB000116710
Last Modified Date : 02/10/2018
Show Technical Document Details
Issue:
After enabling CA APM Introscope Agent 10.2.0.27 to monitor our IBM WebSphere Portal 8.5 environment (with WAS 8.5.5.11 and IBM Java 7.1.3.60 / Suse Linux Enterprise Server 12), we experiencing very high CPU usage in production due to the Introscope agent, thread dumps indicates that the high CPU symptom leading to the heartbeat of the agent is caused by the Java Agent.
Environment:
Any Java agent version
Cause:
APM Java agent uses PMIRegistry API to read PMI module data. The dump points that PMIRegistry.get() method taking more CPU. 
The issue is not related to a bug in the  agent code, Agent uses PMIRegistry API and Webshpere API takes more CPU especially java/util/concurrent/ConcurrentHashMap$Segment.get method of JDK. 

Below thread to illustrate the situation:

"Agent Heartbeat" J9VMThread:0x0000000002146600, j9thread_t:0x00007F8CA0D13EA0, java/lang/Thread:0x0000000100CEC318, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0xF, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x2D1A, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00007F8C98D42000, to:0x00007F8C98D83000, size:0x41000)
3XMCPUTIME               CPU usage total: 19053.985865272 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=822757336 (0x310A47D8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/util/concurrent/ConcurrentHashMap$Segment.get(ConcurrentHashMap.java:443(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ConcurrentHashMap.get(ConcurrentHashMap.java:891(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.lookup(ObjectStreamClass.java:356(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/ArrayList.writeObject(ArrayList.java:754(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor17.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1586(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1547(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/HashMap.writeObject(HashMap.java:1141(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor18.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at java/util/Hashtable.writeObject(Hashtable.java:1042(Compiled Code))
4XESTACKTRACE                at sun/reflect/GeneratedMethodAccessor15.invoke(Bytecode PC:40(Compiled Code))
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56(Compiled Code))
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
4XESTACKTRACE                at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1110(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1534(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1470(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1206(Compiled Code))
4XESTACKTRACE                at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:368(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/session/WsSessionContextMBean.getSessionsInfo(WsSessionContextMBean.java:103(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/session/PMISessionStatistics$1.getRuntimeInfo(PMISessionStatistics.java:68(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/modules/ServletSessionsModule$SessionObjSize.updateStatistic(ServletSessionsModule.java:486(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/data/SpdStatExternal.updateExternal(SpdStatExternal.java:77(Compiled Code))
4XESTACKTRACE                at com/ibm/websphere/pmi/server/PmiAbstractModule.update(PmiAbstractModule.java:1395(Compiled Code))
4XESTACKTRACE                at com/ibm/websphere/pmi/server/PmiAbstractModule.updateStatistics(PmiAbstractModule.java:1418(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.updateStatisticsForStatsTree(ModuleItem.java:588(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.updateStatisticsForStatsTree(ModuleItem.java:599(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/ModuleItem.getStats(ModuleItem.java:717(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/PmiRegistry.getStats(PmiRegistry.java:930(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/pmi/server/PmiRegistry.get(PmiRegistry.java:885(Compiled Code))
4XESTACKTRACE                at com/wily/introscope/agent/pmi/PMIDataPoller$PMIModule.pollPMIForUpdates(PMIDataPoller.java:334(Compiled Code))
4XESTACKTRACE                at com/wily/introscope/agent/pmi/PMIDataPoller.ITimestampedRunnable_execute(PMIDataPoller.java:201(Compiled Code))
..
Resolution:
On the Java Agent side:

a) Disable PMI feature: introscope.agent.pmi.enable=fase. 
You need to restart the JVM/Appserver

b) Keep PMI feature enabled but reduce PMI collection
Disable all Websphere PMI Modules in the IntroscopeAgent.profile and enable them 1 by 1 in order to identify the best settings, for example:
..
introscope.agent.pmi.enable.threadPoolModule=false 
introscope.agent.pmi.enable.servletSessionsModule=false 
introscope.agent.pmi.enable.connectionPoolModule=false 
introscope.agent.pmi.enable.j2cModule=true 
..
See: https://docops.ca.com/ca-apm/10-7/en/implementing-agents/java-agent/java-agent-properties#JavaAgentProperties-WebSpherePMIProperties 


On the Appserver side:

a) Upgrade the Appserver JAVA version

b) The WAS Admin Console provides more granular control over which PMI metrics are reported for all consumers including Introscope 
https://docops.ca.com/ca-apm/10-7/en/implementing-agents/java-agent/install-the-java-agent/configure-application-server-to-use-the-java-agent/websphere-application-server#WebSphereApplicationServer-EnabletheCollectionofIBMWebSpherePMIMetrics 
Additional Information:
All APM compatibility guides are available from:
https://support.ca.com/us/product-content/status/compatibility-matrix/application-performance-management-compatibility-guide.html
or
https://pcm.ca.com/